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ABSTRACT 

This  paper  describes  the  Logic,  Assumptions  and  Risks  Framework  (LARF)  developed  by 
DSTO  to  establish  a  mechanism  to  improve  the  analysis,  transparency  and  decision  paths  that 
underpin  planning  and  evaluation.  The  challenge  was  to  develop  an  approach  based  on 
established  theory,  but  tailored  for  Defence-specific  requirements,  particularly  for  campaign 
planning  and  assessment  activities.  This  framework  has  been  validated  through  a  range  of 
trials  on  critical  campaign  functions  and  has  proven  to  be  effective.  Consequently,  it  is 
assessed  as  being  well-suited  to  the  current  and  future  Defence  decision-making  environment, 
and  is  likely  to  continue  to  be  applied  as  part  of  DSTO's  support  in  these  areas. 


RELEASE  LIMITATION 

Approved  for  public  release 


UNCLASSIFIED 


UNCLASSIFIED 


Published  by 

Joint  Operations  Division 

DSTO  Defence  Science  and  Technology  Organisation 
PO  Box  1500 

Canberra  ACT  2600  Australia 

Telephone:  1300  DEFENCE 
Fax:  (08)  7389  6567 

©  Commonwealth  of  Australia  2013 

AR-015-610 

May  2013 


APPROVED  FOR  PUBLIC  RELEASE 


UNCLASSIFIED 


UNCLASSIFIED 


Logic  Assumptions  and  Risks  Framework  Applied  to 
Defence  Campaign  Planning  and  Evaluation 


Executive  Summary 


Evaluating  the  progress  of  ADF  operations  has  come  into  focus  recently  in  the 
Afghanistan,  East  Timor  and  Solomon  Islands  campaigns.  In  response,  the  Operational 
Planning  and  Evaluation  Team  within  DSTO  Operations  Support  Centre  (DOSC) 
began  in  mid-2010  to  design  and  implement  a  repeatable  process  for  conducting 
campaign-level  monitoring  and  evaluation,  to  support  planning.  The  result  was  the 
development  in  collaboration  with  the  Plans  Branch  of  Headquarters  Joint  Operations 
Command  (HQJOC),  of  the  Campaign  Assessment  (CA)  process,  which  includes 
techniques  adapted  from  methodologies  in  other  domains  to  address  the  challenges  of 
evaluation  in  complex  conflict  or  post-crisis  contexts. 

The  problem  for  analysts  in  Defence  is  that  it  is  difficult  to  assess  the  success  of  a 
campaign  if  the  campaign's  plan  does  not  adequately  articulate  the  intent,  scope  and 
expected  outcomes  of  operational  activity.  Whilst  all  plans  state  objectives  and  various 
levels  of  goals,  it  is  often  not  clear  why  a  series  of  objectives  is  expected  to  lead  to  an 
overall  positive  outcome,  or  to  what  degree  and  in  exactly  what  form  a  goal  must  be 
achieved  to  constitute  'success'.  DSTO  evaluation  analysts  consistently  find  that  these 
explanations  of  linkage  between  intent  and  expected  outcome  are  missing  from 
campaign  plans. 

In  order  to  address  these  problems,  this  research  has  three  aims.  The  first  is  to  identify 
priority  information  requirements,  by  combining  insights  from  practical  experience  in 
CA  cycles  with  a  literature  review  of  planning  and  evaluation  theories.  The  second  aim 
of  the  research  is  to  investigate  whether  it  is  possible  to  develop  a  framework  that  can 
be  applied  to  the  CA  process  to  draw  out  the  information  components  that  have  been 
identified  as  priorities.  The  third  aim  of  the  research  is  to  consider  whether  the 
framework  developed  can  be  applied  to  drafting  of  new  plans  and  reviews  of  existing 
ones,  so  that  priority  information  components  are  clearly  articulated  in  future  plans. 

The  research  identifies  that  the  priority  information  requirements  for  enhanced 
campaign-level  planning  and  evaluation  are  logic,  assumptions  and  risks.  This  is 
supported  by  review  of  several  dominant  theories  in  the  disciplines  of  evaluation 
science,  cognitive  science,  organisational  psychology,  and  decision  support 
engineering;  namely  Theory  of  Change,  Program  Evaluation  and  Cognitive  Mapping. 
Each  provides  guidance  for  scientifically  considering  a  problem,  but  not  the  actual 
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techniques  for  applying  that  problem  structure  throughout  the  phases  of  planning  and 
evaluation.  Therefore,  it  is  concluded  that  the  theoretical  insights  require  a  framework 
for  practical  application,  and  although  a  large  variety  of  frameworks  and  techniques 
have  been  developed  by  other  agencies  to  apply  these  theories,  none  entirely  meet  the 
specific  needs  of  Defence.  Consequently  a  bespoke  framework  blending  the  advantages 
of  a  variety  of  approaches  is  required,  and  the  Logic  Assumptions  and  Risks 
Framework  (LARF)  has  been  developed  to  fill  this  methodological  gap.  The  LARF 
provides  a  mechanism  for  systematically  eliciting  the  logic,  assumptions  and  risks  in 
campaign  planning  and  evaluation  with  some  assurance  of  comprehensive 
consideration  across  all  components  of  a  plan.  It  then  enables  aggregation  of  those 
assumptions  and  risks  and  visibility  of  any  primary  risks  that  are  common  across 
multiple  Effects,  which  can  be  highlighted  and  prioritised  for  monitoring  or 
prevention. 

Another  key  characteristic  of  the  LARF  is  the  scalability  of  its  structure,  depending  on 
what  users  are  aiming  for  and  to  what  extent  a  plan  or  evaluation  framework  can  be 
altered.  Both  Theory  of  Change  and  Program  Theory  allow  for  either  micro-  or  macro¬ 
theories  to  be  drawn  out  and  analysed.  By  synthesising  both  of  these  theories  into  the 
LARF,  it  can  be  used  to  gain  insight  into  either  the  detail  of  how  activities  and 
component  parts  of  a  plan  are  expected  to  work  (ie.  the  micro-theory),  or  to  gain 
insight  into  how  the  sum  of  the  components  is  assumed  to  culminate  in  the 
achievement  of  some  form  of  change  (ie.  macro-theory). 

The  LARF  is  assessed  through  trials  on  four  critical  campaign  functions  in  three 
different  Defence  operations,  and  proves  to  be  an  effective  tool  in  each  instance,  with  a 
number  of  unexpected  advantages.  In  summary  to  date,  the  LARF  has  been  used  as: 

•  an  initial  planning  tool  -  to  test  viability  of  objectives,  draft  underlying  detail  of 
those  objectives,  and  concept  of  how  their  achievement  will  be  assessed; 

•  a  group  learning  exercise  for  building  situational  awareness  -  the  basis  of  a  forum 
for  exchange  of  ideas  and  knowledge,  for  experienced  and  inexperienced.  Also 
served  as  an  effective  team-building  exercise  which  revealed  individual's  areas  of 
expertise,  strengths  and  weaknesses; 

•  a  structured  format  for  revision  of  an  existing  plan  and  evaluation  framework; 

•  a  tool  for  generation  of  a  set  of  risks  and  assumptions  to  be  added  as  indicators  to 
an  existing  set  of  measures  of  effectiveness. 

These  trials  demonstrate  that  the  LARF  offers  planners  and  evaluators  the  ability  to 
better  identify  the  relationships  between  action,  results,  risks  and  opportunities.  It 
enables  planners  to  be  more  responsive  and  adaptive  to  changes  in  the  operating 
environment.  By  making  the  campaign's  priorities  transparent  it  facilitates  review  of 
efficient  use  of  resources  and  increases  the  likelihood  of  detecting  and  responding  to 
obstacles.  It  has  been  found  to  distinguish  between  design  or  implementation  flaws  as 
the  reason  for  a  lack  of  progress  towards  defined  objectives  (or  the  reason  for 
progress);  and  to  assist  in  identifying  indicators  and  metrics  for  measuring  progress. 
The  adaptability  of  the  LARF  and  its  suitability  for  rapid  implementation,  without  a 
significant  resource  liability,  makes  the  LARF  well-suited  to  meeting  current  Defence 
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planning  and  evaluation  requirements,  and  it  is  simple  and  practical  enough  to  be 
adapted  for  a  wide  variety  of  future  functions. 

There  remains  significant  scope  for  further  development,  validation  and  application  of 
the  LARF,  particularly  in  the  use  of  data  elicited  in  the  LARF  matrix  as  a  reference  for 
tracking  whether  planning  logic  and  assumptions  were  correct  as  part  of  post- 
operational  evaluations.  However,  research  to  date  has  produced  a  practical  and 
immediately  applicable  framework. 
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1.1  Background 

Recent  Australian  Defence  Force  (ADF)  operations  in  Afghanistan,  East  Timor  and  the 
Solomon  Islands  have  highlighted  that  existing  methods  for  planning,  monitoring  and 
evaluating  activities  in  other  domains  are  not  well  suited  to  Defence's  unique  requirements. 
Defence  decision  makers  are  required  to  translate  Government  direction  into  plans,  to  monitor 
progress,  and  to  choose  appropriate  courses  of  action  to  maintain  progress  in  a  highly 
uncertain  environment.  In  any  operation.  Defence  is  not  an  isolated  actor,  and  there  are  many 
influences  on  the  likelihood  of  success  or  failure  which  have  nothing  to  do  with  military 
activities.  Contributing  factors  and  causal  dynamics  are  not  always  apparent  or  quantifiable, 
making  industrial  planning  and  evaluation  models  of  inputs,  outputs  and  milestones 
inappropriate.  The  extent  to  which  desired  conditions  are  emerging  in  the  operating 
environment,  whether  the  presence  of  these  conditions  is  likely  to  continue,  and  the  extent  to 
which  the  ADF  is  able  to  influence  them,  requires  consideration  of  contextual  information 
from  the  broader  operating  environment.  Traditional  Defence  campaign  assessments  are  not 
designed  to  take  account  of  this  level  of  contextual  complexity.  As  a  consequence,  they  are  not 
able  to  report  on  progress  with  transparent,  repeatable  analysis  frameworks  which  provide 
assurance  that  assessments  are  both  comprehensive  and  evidence-based. 

In  recognition  of  this,  the  Operational  Planning  and  Evaluation  Team  within  the  Defence 
Science  and  Technology  Organisation's  (DSTO)  Operations  Support  Centre  (DOSC)  was 
tasked  in  mid-2010  to  design  and  implement  a  repeatable  process  for  conducting  campaign- 
level  monitoring  and  evaluation,  to  support  planning.  The  result  was  the  development  and 
institutionalisation  of  the  Campaign  Assessment  (CA)  process.  This  included  techniques 
adapted  from  methodologies  in  other  domains  to  address  the  challenges  of  evaluation  in 
complex  conflict  or  post-crisis  contexts. 

The  problem  for  analysts  is  that  assessing  the  success  of  a  campaign  is  problematic  if  the 
campaign's  plan  does  not  adequately  articulate  the  intent,  scope  and  expected  outcomes  of 
operational  activity  to  stakeholders.  Whilst  all  plans  do  state  objectives  and  various  levels  of 
goals,  it  is  often  not  clear  why  a  series  of  objectives  is  expected  to  lead  to  an  overall  positive 
outcome,  or  to  what  degree  and  in  exactly  what  form  a  goal  must  be  achieved  to  constitute 
'success' 1 .  These  explanations  of  linkage  between  intent  and  expected  outcome  are  the  form  of 
logic  that  DSTO  evaluation  analysts  consistently  find  missing  from  campaign  plans. 

Given  these  problems,  research  was  conducted  to  address  the  question  of  precisely  what 
information  components  of  a  plan  are  of  primary  importance  for  CA?  A  second  question  that 
also  arises  is,  once  these  priority  information  components  are  identified,  how  can  we  derive 
them  from  any  existing  campaign  plan?  In  the  next  section,  the  specific  aims  of  the  research 
program  designed  to  answer  these  questions  is  detailed. 


1  This  concept  is  further  discussed  in  Chapter  3:  Literature  Review. 
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1.2  Aims 

The  first  aim  of  the  research  is  to  highlight  the  priority  information  components  of  a  plan  that 
enable  CA.  The  definition  of  what  constitutes  priority  information  requirements  will  be 
determined  by  combining  insights  from  practical  experience  in  CA  cycles2  with  a  literature 
review  (in  Chapter  3)  of  planning  and  evaluation  theories. 

The  second  aim  of  the  research  is  to  investigate  whether  it  is  possible  to  develop  a  framework 
that  can  be  applied  to  the  CA  process  to  draw  out  the  information  components  that  have  been 
identified  as  priorities.  Given  that  CA  analysis  teams  have  an  on-going  requirement  for 
techniques  for  extracting  a  greater  depth  of  understanding  of  respective  campaign  plans,  any 
mechanism  for  rapidly  identifying  priority  information  components  in  large,  complex  plans  is 
desirable.  As  discussed  later,  greater  depth  of  understanding  enables  analysts  to  evaluate  the 
plan  within  the  appropriate  context,  to  identify  a  greater  number  of  pertinent  indicators  of 
progress,  and  to  identify  how  these  can  be  most  effectively  monitored  (Weiss  1997). 

The  third  and  final  aim  of  the  research  is  to  consider  whether  the  framework  developed  can  be 
applied  to  drafting  of  new  plans  and  reviews  of  existing  ones,  so  that  priority  information 
components  are  clearly  articulated  in  future  plans.  This  is  intended  to  enhance  the  quality  of 
plans  in  general,  as  well  as  facilitating  the  CA  process  in  future. 

1.3  Scope 

Research  to  achieve  these  aims  is  best  carried  out  within  the  scope  of  the  following  series  of 
six  steps: 

1.  Establish  the  requirements  for  the  research,  develop  clear  definitions  of  the  main  concepts 
and  identify  the  constraints  for  the  research.  This  will  include  consideration  of  the 
information  gaps  and  information  requirements  in  line  with  the  first  aim  of  identifying 
priority  information  components  to  enable  CA. 

2.  Review  the  literature  to  first  identify  theories  that  identify  priority  information 
requirements  for  planning  and  evaluation  to  contribute  to  the  first  aim  and  second,  to 
identify  hypothesised  techniques  for  eliciting  and  analysing  these  information 
requirements.  The  step  will  support  achievement  of  the  second  aim. 

3.  Develop  a  proposed  solution  that  synthesises  theoretical  constructs  and  practical 
considerations.  Building  upon  the  previous  step  in  which  the  viability  of  a  framework  as  a 
solution  is  considered,  this  step  will  achieve  the  second  aim  by  creating  a  single 
framework  which  is  optimised  for  Defence  operational  requirements. 

4.  Test  the  proposed  framework  through  trial  applications  in  a  range  of  real-time  operational 
planning  and  evaluation  tasks.  This  step  will  consolidate  achievement  of  the  second  aim, 
through  analysis  of  the  benefits  and  weaknesses  of  the  framework  revealed  by  the  trials.  It 
will  also  support  the  third  aim  by  considering  the  applicability  of  the  framework  to 
planning  processes  by  including  trials  on  planning  activities.  From  this  we  are  able  to 
draw  conclusions  on  the  viability  of  the  research  and  the  need  for  further  development  or 
testing  of  the  framework. 


2  Evidence  and  discussion  of  all  CA  cycles  to  date  are  contained  in  classified  DSTO  reports  on  support  to 
operations  in  Afghanistan,  East  Timor  and  The  Solomon  Islands,  which  can  not  be  referenced  here.  However,  they 
will  be  described  in  generic  terms  in  case  studies  under  Chapter  5. 
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5.  Integrate  trial  findings  into  further  research  and  development.  Identify  additional  trial 
applications  and  formal  validation  (including  capturing  user  feedback). 

6.  Finalise  the  framework  through  endorsed  integration  into  Defence  guidance  documents 
and  user  education  programs. 

1.4  Report  Structure 

Chapter  1  is  this  chapter.  In  Chapter  2  we  will  establish  a  clear  explanation  of  the  research 
context,  by  defining  and  discussing  key  elements,  characteristics  and  constraints  of  the 
problem  context.  Chapter  2  is  the  result  of  Step  1  of  the  scope  of  the  research.  Chapter  3,  the 
literature  review,  will  highlight  the  main  theories  that  are  relevant  to  the  research  problem 
and  represents  Step  2  of  the  research).  The  strengths  and  weakness  of  each  of  the  theories  is 
identified  and  a  summary  of  the  main  points  relating  to  the  research  is  presented  (Step  3). 
Chapter  4  continues  with  a  description  of  the  method  for  developing  a  Defence-tailored 
framework  (Step  3).  The  research  method  used  to  develop  and  trial  the  framework  is  then 
presented.  This  will  include  the  criteria  for  rating  the  success  of  trials  (Step  4.)  Chapter  5  (as 
part  of  Step  4),  will  provide  descriptions  of  a  number  of  situations  in  which  the  framework 
was  used.  The  results  from  these  use-cases  will  then  be  discussed  .  Chapter  6  reflects  on  the 
implications  of  these  findings  for  further  development  of  the  framework.  Finally,  Chapter  7 
will  draw  conclusions  on  the  extent  to  which  the  research  aims  have  been  met.  Indications 
from  the  research  to  date  about  the  viability  of  the  framework  will  be  identified.  This  final 
Chapter  will  document  the  completion  of  Step  4,  and  will  identify  future  research,  (Steps  5 
and  6). 

1.5  Conclusion 

This  chapter  has  explained  the  origin  of  the  DSTO's  support  to  Australian  Defence 
operational-level  planning  and  evaluation,  and  highlighted  some  of  the  persistent  challenges 
faced  in  providing  analytical  support  to  these  tasks.  Investigating  options  to  overcome  these 
challenges  has  become  the  aim  of  a  research  team  within  Operational  Planning  and 
Evaluation  Team,  and  has  led  to  the  development  of  the  framework  described  in  the 
remainder  of  this  report.  In  the  next  chapter  we  will  explore  the  operational  domain  that 
shaped  the  research,  by  providing  definitions  of  key  aspects  and  the  constraints  of  the  context 
of  application. 


2  Operational  Domain 

The  previous  chapter  described  the  background  in  which  the  requirement  for  the  DSTO's 
research  and  support  emerged,  and  the  improvements  that  this  research  aims  to  make.  This 
chapter  will  further  define  the  context  of  the  task  to  enable  a  more  detailed  discussion  and 
understanding  of  the  boundaries  of  the  research  team's  contribution.  To  do  this,  we  will  first 
define  the  key  components  of  Campaign  Plans  and  Campaign  Assessment,  identifying  the 
current  limitations  and  implications  of  each  for  analysts.  We  will  then  consider  the  practical 
constraints  pertaining  to  the  Defence  operational  context  of  application.  These  constraints  will 
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form  the  basis  for  the  criteria  set  out  in  Chapter  4  for  assessing  the  Defence-specific 
practicality  of  research  outcomes. 


2.1  Campaign  Plans 


As  a  long-term  planning  instrument,  campaign  plans  are  designed  to  articulate  Defence's 
intended  contribution  to  a  Government  initiative,  and  translate  this  intent  into  a  cohesive 
program  of  military  activity  that  will  align  with  Government  strategy  from  start  to  finish.  In  a 
Whole  of  Government  context,  the  campaign  plan  can  be  thought  of  as  a  program  planning 
structure  in  which  the  military  component  of  national  strategic  objectives  is  defined,  and  this 
strategic  military  component  is  broken  down  into  broad  Decisive  Conditions. 


Whole  of  Government 


ADF 


Figure  1:  Example  of  an  ADF  Campaign  Plan  mapped  hierarchically  from  National  Strategy,  to  the 
ADF  End  State,  to  Decisive  Conditions  and  downwards  to  Effects 

Figure  1  shows  that  ADF  campaign  plan  Decisive  Conditions  are  comprised  of  high-level 
objectives  which  in  turn,  are  broken  down  into  actionable  Effects  (or  progress  milestones)  that 
will  lead  to  the  achievement  of  those  objectives.  Effects  define  the  boundaries  for  the  range  of 
activities  that  the  Commander  Joint  Operations  (CJOPS)  plans  to  undertake  and  describes  the 
scope  of  influence  he  expects  to  have  in  the  area  of  operations  (Hickman  2012).  It  is  important 
to  note  that  Effects  are  written  as  outcomes  rather  than  actions,  and  they  each  describe  a 
condition  or  change  in  condition  which  is  expected  to  be  achieved  by  associated  tasks  and 
activities.  In  this  way,  an  Effects-based  military  campaign  is  what  Williams  and  Morris  (2009) 
define  as  a  sequence  of  hypotheses,  referred  to  in  evaluation  practice  as  a  normative  program 
theory,  or  theory  of  change. 

The  quality  and  longevity  of  a  plan  is  contingent  on  articulation  of  clear  links  between  the 
elements  of  the  hierarchy.  The  comprehensiveness  and  cohesiveness  of  a  campaign  plan  is 
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demonstrated  by  the  extent  to  which  the  ultimate  Effects  can  be  aggregated  upwards  to 
logically  lead  to  the  emergence  of  the  Decisive  Conditions,  which  can  subsequently  be  clearly 
understood  to  be  pre-conditions  for  achievement  of  the  ADF  End  State.  The  extent  to  which 
these  linkages  should  be  detailed  in  campaign  plans  is  not  prescribed  by  Defence  doctrine. 
However,  for  the  purpose  of  this  research,  they  are  considered  to  be  priority  information 
requirements  for  planning  and  CA,  and  will  be  referred  to  in  further  discussion  as  'logic', 
'assumptions'  and  'risks'. 

The  DSTO  team's  exposure  to  campaign  planning  and  CA  since  20103  has  indicated  that  the 
logic  for  embarking  on  a  given  course  of  action  is  often  assumed  to  be  self-evident,  and  the 
planner's  perception  of  the  operating  environment  is  assumed  to  be  accurate  and  enduring. 
However,  as  the  global  history  of  program  failures  and  unsuccessful  military  operations  have 
demonstrated4,  both  logic  and  assumptions  can  be  flawed,  or  at  the  very  least  need  to  be 
revised  to  align  with  an  evolving  reality.  Without  articulation  of  logical  links,  the  plan  cannot 
be  accurately  interpreted  into  action.  Without  articulated  logic  and  assumptions  as  a  reference 
point,  plans  cannot  be  modified  in  response  to  shifts  in  the  complex  operating  environment, 
without  risk  of  losing  alignment  with  original  intent  and  national  strategy.  In  this  instance,  it 
also  becomes  difficult  to  identify  indicators  of  progress  along,  or  divergences  from,  intended 
paths,  when  those  paths  are  not  clearly  articulated.  This  is  a  significant  obstacle  for  effective 
evaluation. 

Lack  of  articulated  logic  and  assumptions  has  been  a  significant  issue  during  the  DSTO's 
support  to  drafting  and  updating  of  campaign  plans.  During  these  planning  forums, 
divergence  in  the  perspectives  of  various  planning  staff  became  difficult  to  resolve  if 
underlying  logic  and  assumptions  were  not  surfaced  and  addressed.  It  is  also  evident  that 
entire  plans  can  be  developed,  based  on  a  set  of  assumptions  that  are  never  articulated.  This 
significantly  impacts  upon  the  degree  to  which  the  intent  of  the  plan  is  accurately  understood, 
and  translated  into  appropriate  operational  action. 

2.2  Campaign  Assessment 

Campaign  Assessment  (CA)  is  the  means  by  which  the  ADF  is  able  to  review  progress 
towards  the  goals  defined  in  a  campaign  plan.  Progress  is  defined  in  terms  of  likelihood  of 
success,  as  well  as  resources  and  time  required  for  success.  Assessments  are  made  at  regular 


3  To  date,  this  comprises  more  than  eight  CA  cycles  and  planning  reviews  for  Op  SLIPPER 
(Afghanistan),  Op  ASTUTE  (East  Timor)  and  Op  ANODE  (The  Solomon  Islands),  in  addition  to  the 
plan  drafting  process  for  a  non  operational  Regional-engagement  campaign  plan.  References  for  these 
reports  are  Classified. 

4  Examples  include  those  detailed  in:  K.  Betts  "Analysis,  War  and  Decision:  Why  Intelligence  failures 
are  inevitable",  in  World  Politics,  Princeton  University  Press,  1978;  T.  McConville,  "The  War  on 
Terrorism:  A  New  Classic  in  Groupthink",  in  Defence  Management  in  Uncertain  Times,  Cranfield 
Management  Series  No.  3,  USA,  2003;  F.  Sheils,  "Prevantable  Disasters:  Why  Governments  Fail", 
Rowman  and  Littlefield  Publishers  Inc,  USA,  1991;  E.  Larson,  D.  Orletsky  and  K.  Leuschner,  "Defense 
Planning  in  a  Decade  of  Change",  RAND  Project  Airforce,  2001;  S.  Casey  and  J  Wright  (eds)  "Mental 
Maps  in  the  Era  of  Two  World  Wars",  Palgrave  Macmillan,  2008;  United  States  General  Accounting 
Office  "Foreign  Assistance:  Observations  on  Post-Conflict  Assistance  in  Bosnia,  Kosovo  and 
Afghanistan"  GAO-03-980T,  July  2003. 
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intervals  (i.e.  quarterly  or  bi-annually)  comparing  actual  progress  with  intended  progress  (i.e. 
time,  resources  and  milestones  set  out  in  the  plan).  The  reliability  and  insights  of  reports 
generated  through  CA  process  developed  by  the  DSTO  has  resulted  in  the  institutionalisation 
of  the  process  as  part  of  HQJOC  planning,  decision-making  and  senior  command  advice. 

Specifically,  the  DSTO's  support  to  the  CA  process  is  intended  to  assist  decision  makers  at  all 
levels  to: 

•  maintain  visibility  of  whether  the  original  Commander's  intent  is  being  sustained; 

•  identify  whether  resources  are  being  put  to  best  effect  (i.e.  whether  gains  are  being 
built  upon,  and  where  less  effective  activities  can  be  discontinued);  and 

•  where  necessary,  adapt  initial  plans  and  goals  to  align  with  changes  in  the 
environment. 

CA  must  look  beyond  ADF  inputs  and  outputs,  and  cannot  be  conducted  by  simply  reviewing 
the  performance  of  the  ADF  on  operations  in  order  to  make  assessments  of  actual  progress 
towards  Decisive  Conditions,  as  defined  in  DSTO  guidance  on  support  to  campaign 
assessment  (Flickman  2012).  This  requires  the  construction  of  a  rich  picture  which 
incorporates  as  many  contextual  variables  as  possible,  including  the  influence  of  all  non- ADF 
players  in  the  operational  space.  This  has  been  achieved  by  incorporating  elements  of  civilian 
evaluation  practice,  into  the  CA  process,  to  move  it  beyond  simple  battle  damage  assessments, 
and  ensure  that  the  analysis  contained  in  a  CA  is  verified  with  as  much  evidence  as  possible, 
and  is  a  reliable  source  of  situational  awareness  for  senior  ADF  decision  makers. 

As  will  be  discussed  as  part  of  the  literature  review  in  the  next  chapter,  many  of  the  civilian 
approaches  to  planning  and  evaluation  require  visibility  and  analysis  of  underlying  logic,  the 
assumptions  upon  which  that  logic  is  based,  and  the  risks  identified  and  accepted  within  the 
plan.  Surfacing  of  the  logic,  assumptions  and  risks  of  campaign  plans  is  essential  for  analysts 
to  recognise  what  variables  and  evidence  are  pertinent  to  making  credible  assessments.  Our 
experience  in  successive  CA  cycles5  has  been  that  while  they  have  highlighted  some  of  these 
information  elements,  for  some  components  of  the  campaign,  their  consideration  has  been 
random  and  inconsistent,  rather  than  by  design.  There  has  been  no  consistent  mechanism  for 
identifying  the  logic  and  assumptions  of  a  plan,  or  for  monitoring  whether  they  are  flawed 
and  are  putting  the  plan  at  risk.  CAs  have  instead  focused  on  measuring  positive  or  negative 
indicators  of  progress. 

The  veracity  of  a  plan  can  only  be  analysed  if  it's  underpinning  logic,  assumptions  and  risks 
are  transparent,  and  can  be  compared  with  emerging  reality.  Indicators  of  success  or  failure 
are  valuable  for  reporting  on  progress  trends  to  date,  but  are  of  less  value  for  providing 
assurance  of  why  current  trends  should,  or  shouldn't,  be  expected  to  continue.  Assessments  of 
what  is  likely  to  happen  in  future  must  be  aligned  with  the  assumptions  of  the  campaign  plan. 
It  is  these  assessments  that  are  of  greatest  value  in  informing  decisions  on  what  action  should 
be  taken  next.  Therefore,  when  seeking  to  produce  CAs  that  assist  decision  makers  in  the 


5  Evidence  and  discussion  of  all  CA  cycles  to  date  are  contained  in  classified  DSTO  Technical  Reports  on  support 
to  operations  in  Afghanistan,  East  Timor  and  The  Solomon  Islands,  which  can  not  be  referenced  here.  However, 
they  will  be  described  in  generic  terms  in  case  studies  under  Chapter  5. 
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ways  listed  above,  assessments  still  need  to  report  on  progress  trends  to  date,  but  those  trends 
must  be  analysed  in  terms  of  what  they  mean  for  the  veracity  of  the  plan  going  forward. 

This  is  not  to  suggest  that  all  initial  plans  are  flawed,  but  rather  recognises  that  in  a  complex 
environment  that  involves  highly  variable  inputs,  it  should  be  expected  that  the  logic  and 
assumptions  upon  which  initial  plans  were  based  will  need  to  be  corrected  as  the  operating 
environment  evolves.  Defence  operations  occur  within  an  environment  where  multiple  actors, 
timeframes,  priorities,  and  mechanisms  for  pursuing  a  course  of  action,  all  intersect,  and  no 
aspect  of  the  environment  can  be  expected  to  be  static.  Therefore,  planning  must  be  iterative 
and  responsive,  and  CAs  must  be  able  to  inform  iterative  planning  by  including  systematic 
consideration  of  the  veracity  of  the  plan's  logic,  assumptions  and  identified  risks,  in  light  of 
emerging  evidence. 

2.3  Practical  constraints 

As  suggested  in  Section  1.1,  Defence  has  a  number  of  unique  characteristics  that  limit  the 
applicability  of  existing  techniques  for  monitoring  and  evaluation  from  other  domains  such  as 
industry  and  development  agencies.  These  characteristics  serve  as  constraints  on  the  options 
for  practical  research  outcomes,  and  these  are  briefly  described  now.  There  are  four  major 
practical  constraints:  ADF  resource  limitations,  the  complexity  of  campaign  environments, 
data  collection  limitations  and  requirements  of  decision  makers.  These  constraints  are 
described  in  more  detail  in  the  following  paragraphs. 

The  first  constraint  is  the  lack  of  ADF  resources  to  dedicate  to  any  component  of  campaign 
planning  and  evaluation.  In  contrast  some  other  international  agencies  or  corporations  in 
Australia  and  overseas,  the  ADF  does  not  have  a  large  team  dedicated  exclusively  to 
conducting  program-level  evaluations,  and  interfacing  with  planning  staff.  As  a  result,  any 
practical  addition  to  ADF  planning  and  evaluation  processes  must  require  a  minimum  of  staff 
and  time  investment.  As  will  be  elaborated  on  in  Chapter  3,  many  existing  program  planning 
frameworks  require  a  sizeable  planning  team  to  conduct  detailed  investigations  as  inputs  to 
the  process.  Whilst  the  commitment  of  this  scale  of  resources  for  a  rigorous  and 
comprehensive  planning  process  is  desirable,  it  is  not  within  the  capacity  or  timeframes  of  the 
current  ADF  operational  planning  structure. 

Secondly,  as  already  highlighted  in  the  definition  of  CA,  a  solution  is  sought  that  is  applicable 
in  the  ADF's  complex  operating  environments,  characterised  by  high  degrees  of  uncertainty. 
There  is  potential  in  the  Defence  context  for  action  to  be  paralysed  by  uncertainty.  Whilst 
planning  of  action  and  assessment  of  the  likelihood  of  success  is  undoubtedly  enhanced  by 
greater  degrees  of  contextual  understanding,  there  must  be  a  limit  to  the  time  and  resources 
expended  on  establishing  this  understanding  prior  to  acting.  Therefore,  an  appropriate 
solution  must  not  be  dependent  on  certainty  about  the  variable  of  the  context  within  which 
the  plan  will  be  executed,  such  as  inputs,  outputs,  and  tangible  goals.  Again  as  already 
suggested  under  the  definition  of  CA  above,  in  such  uncertainty,  decisions  will  be  iterative 
and  emergent,  and  will  require  planning  and  evaluation  tools  that  are  repeatable  and  equally 
adaptive. 
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A  third  constraint  is  the  likelihood  of  gathering  data  for  planning  or  evaluation  in  an 
environment  which  is  subject  to  conflict  or  crisis;  referred  to  as  a  non-permissive  environment  by 
the  ADF.  The  likelihood  of  operating  in  a  non-permissive  environment  significantly  constrains 
the  practicality  of  conducting  consultation  with  stakeholders,  and  target  populations  of  a  plan, 
to  determine  goals  and  identify  how  these  goals  will  be  achieved  and  evaluated.  When 
operating  in  environments  where  there  may  be  adversaries,  or  where  there  is  potential  for  the 
situation  to  deteriorate  if  some  forms  of  information  are  made  available,  the  scope  for 
collaboration  and  consultation  is  limited.  It  also  often  means  that  planning  is  being  conducted 
in  response  to  a  crisis,  with  a  short  lead  time  for  consultation  and  information  gathering.  This 
is  contrary  to  a  large  portion  of  existing  techniques  for  enhancing  planning  and  evaluation, 
which  advocate  participative  research  (Van  der  Riet  2001)  and  collaborative  evaluation 
wherever  possible.  Again,  while  the  merits  of  high  degrees  of  consultation  and  local 
participation  are  acknowledged,  this  requires  longer  timeframes  and  levels  of  engagement 
that  are  not  currently  considered  viable  for  military  operations  in  post-crisis  or  conflict 
environments. 

The  time  constraints  of  carrying  out  research  concurrently  with  executing  actual  CA  and 
planning  tasks,  meant  that  research  commenced  without  a  series  of  activities  dedicated  to 
elicitation  of  client  requirements,  engagement  of  the  client  in  formal  needs  elicitation 
programs,  or  dedication  of  time  and  resources  to  a  scoping  study.  This  decision  is  reinforced 
by  the  fact  that  the  research  follows  several  years  of  work  within  the  problem  domain,  from 
which  it  is  believed  sufficient  understanding  has  already  been  established. 

Finally,  given  that  the  ADF  is  operating  in  response  to  political  guidance,  decisions  on 
operational  direction  and  scale  may  be  already  set  by  the  time  that  planners  or  evaluators  are 
directed  to  devise  a  way  forward.  Therefore,  it  is  often  likely  that  there  will  be  constraints  on 
the  level  of  planning  or  evaluation  sought  in  the  operational  context.  Techniques  designed  to 
focus  on  'greenfield'  scoping  of  solutions  and  strategies  may  waste  significant  time  analysing 
factors  which  are  not  changeable  within  the  Defence  context. 

These  practical  constraints  form  the  basis  of  a  set  of  criteria  that  will  be  listed  in  Chapter  4,  for 
testing  the  practicality  of  the  framework  developed  here,  which  is  considered  an  important 
component  of  assessing  the  extent  to  which  overall  research  aims  have  been  met. 

2.4  Conclusion 

This  chapter  has  defined  the  context  within  which  research  is  being  conducted.  It  is  apparent 
that  Defence  campaign  planning  combines  the  well  established  ADF  operational  planning 
process  with  the  evolving  mechanisms  for  translating  Government  strategic  intent  into 
Defence  objectives  and  effects.  This  context  has  some  similarities  with  all  program-level 
planning,  including  the  requirement  for  visibility  of  a  plan's  logic,  assumptions  and  risks  for 
analysts  to  recognise  what  variables  and  evidence  are  pertinent  to  making  credible 
assessments.  It  is  also  apparent  that  the  research  context  has  many  unique  constraints  that  are 
problematic  for  the  provision  of  scientific  analysis  support  to  campaign  planning  and 
evaluation  of  plans.  As  highlighted,  one  of  the  implications  of  time,  data  and  resource 
constraints  is  the  lack  of  a  consistent  mechanism  for  identifying  the  logic  and  assumptions  of  a 
plan,  or  for  monitoring  whether  they  are  flawed  and  are  putting  the  plan  at  risk.  The  other 
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constraint  expected  to  impact  upon  the  research  is  variation  in  the  degree  of  influence  or 
change  sought  from  planning  and  evaluation  support,  depending  on  the  level  of  maturity  and 
endorsement  of  a  plan.  The  design  and  testing  of  research  outcomes  can  now  be  understood 
within  the  constraints  of  the  intended  context  of  application.  This  concept  of  what  is  likely  to 
be  relevant  and  beneficial  in  practice  will  be  used  to  assess  the  potential  applicability  of  a 
number  of  theoretical  approaches  in  the  next  chapter. 


3  Literature  Review 


The  previous  chapter  focused  on  the  Defence  context  in  which  the  research  is  situated.  This 
chapter  will  turn  the  focus  outwards,  looking  broadly  at  the  theoretical  constructs  and 
methods  that  have  been  applied  outside  of  Defence  to  achieve  similar  aims.  In  particular,  this 
literature  review  will  show  how  the  information  elements  of  logic,  assumptions  and  risks  are 
elicited  and  analysed  under  several  dominant  theories  in  the  disciplines  of  evaluation  science, 
cognitive  science,  organisational  psychology,  and  decision  support  engineering.  These 
existing  theories  have  potential  to  provide  problem  structures  and  analysis  steps  that  would 
add  transparency  to  the  campaign  planning  process  and  provide  insights  upon  which  more 
effective  evaluation  could  be  based.  The  review  of  theoretical  literature  is  concluded  with  a 
synthesis  of  theories  in  Section  3.5,  which  describes  the  implications  of  each  for  planning  and 
evaluation.  The  review  begins  by  considering  the  theoretical  guidance  that  underpins  all  ADF 
activities  (contained  in  current  Defence  doctrine  pertaining  to  operational  planning  and 
evaluation). 

3.1  ADF  Doctrine 

There  are  currently  several  editions  of  ADF  doctrine  that  pertain  to  operational  planning  and 
evaluation6.  They  aim  to  provide  the  process  tools  to  overcome  the  kind  of  uncertainty  and 
complexity  described  in  the  previous  chapter  as  characterising  the  Defence  operating 
environment.  However,  there  are  several  important  limitations  to  the  extent  to  which  ADF 
doctrine  provides  guidance  for  CA.  For  example,  there  is  a  lack  of  consistent  definitions 
between  different  doctrine  publications.  There  is  also  a  lack  of  detail  and  illustrative  example 
in  guidance  intended  to  cover  the  campaign  level,  which  has  resulted  in  Defence  planners 
interpreting  the  doctrine  in  different  ways  when  developing  campaign  plans  for  recent  ADF 
operations. 

Comparison  of  campaign  plans  for  different  operations  reveals  that  there  is  little  similarity 
between  plans  in  terms  of  how  the  operational  logic  has  been  charted,  how  the  assumptions  of 
the  operation  have  been  revealed  and  tested,  or  how  the  measures  of  operational  progress 
have  been  developed.  This  has  been  problematic  for  the  development  of  a  consistent  approach 
to  effective  assessment  of  campaign  progress  and  effectiveness.  This  was  improved  to  some 
extent,  very  recently  with  a  new  version  of  Australian  Defence  Doctrine  Publication  3.0, 
Campaigns  and  Operations,  released  in  mid  2012.  However,  definitions  and  descriptions  of 


6  Australian  Defence  Doctrine  Publication  5.0,  Joint  Planning  (Provisional),  January  2006;  and  Australian  Defence 
Doctrine  Publication  3.0,  Campaigns  and  Operations,  July  2012;  Australian  Defence  Doctrine  Publication  00.4 — 
Operational  Evaluation,  August  2007. 
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planning  and  evaluation  processes  are  still  not  clearly  aligned  with  what  can  be  seen  to  occur 
in  practice.  Of  most  relevance  for  our  research,  doctrinal  guidance  does  not,  and  is  not 
intended  to,  provide  detail  on  optimal  techniques  for  accomplishing  the  respective  steps 
defined  for  planning  or  evaluation.  Therefore,  it  was  concluded  that  doctrine  directs  (albeit 
inconsistently)  what  needs  to  be  done,  and  the  development  of  concepts  on  how  things  are  to 
be  done  must  be  drawn  from  other  sources. 

3.2  Theory  of  Change 

Theory-based  evaluation  frameworks,  usually  referred  to  as  Theory  of  Change  (ToC),  aim  to 
evaluate  the  action  undertaken  as  part  of  the  program,  simultaneously  with  the  theory  upon 
which  those  actions  and  decisions  were  based.  As  highlighted  by  Bickman  (1987): 

All  too  often  evaluators  focus  on  the  technical  aspects  of  the  evaluation  process  (design  and 
measurement)  and  pay  scant  attention  to  the  program.  A  successful  evaluation  can  produce 
some  short-term  results  and  possibly  local  change,  but  without  considering  the  theory 
underlying  the  program,  neither  evaluators,  program  developers,  nor  program  implemented 
can  hope  to  produce  generalizable  findings.  The  ability  to  generalize  from  a  specific  evaluation 
is  significant  for  policy  makers.  They  need  to  know  whether  the  results  of  a  particular  program 
indicate  that  the  program  or  some  aspects  of  the  program  will  work  in  some  other  setting  and 
time. 

Rather  than  listing  or  quantifying  desirable  outcomes,  theory-based  impact  evaluations 
produce  a  narrative  theory  of  the  change  which  offers  insights  into  why  a  program  is  executed 
in  a  given  manner,  and  subsequently,  why  it  either  generated  or  did  not  generate  the 
anticipated  outcomes  (Weiss  1997).  It  also  seeks  to  provide  answers  as  to  what  extent  external 
factors  contributed  to  these  effects,  and  what  led  to  the  unanticipated  effects  which  were 
observed.  These  insights  suggest  what  remediation  action  should  be  taken  to  correct  or 
reinforce  trends  in  an  on-going  operation,  whilst  also  showing  what  trends  are  context- 
specific,  and  therefore  should  not  be  expected  to  be  replicated  in  the  future. 

All  plans,  programs  and  activities  contain  ToCs,  and  associated  assumptions  about  how  and 
why  action  will  bring  about  change.  One  does  not  have  to  apply  a  ToC  approach  to  create 
such  a  rationale,  because  all  actions  are  undertaken  with  some  concept  of  what  they  are 
expected  to  achieve  and  how.  However,  as  Leeuw  and  Vaessen  (2009)  point  out,  that  rationale 
may  be  either  explicit  or  implied.  Application  of  ToC  simply  ensures  that  planners  explicitly 
articulate  their  intentions,  expectations  and  assumptions,  rather  than  focusing  only  on  what 
will  be  done  and  when.  ToC  is  not  a  prescribed  set  of  steps,  or  methodology,  but  rather  what 
Van  Stolk  (2011)  refers  to  as  a  way  of  thinkingthat  prompts  planners  to  ask  certain  questions 
and  requires  the  articulation  of  answers  in  a  certain  way.  This  is  a  valuable  insight  for  the 
intent  of  problem  structures  and  analysis  techniques  sought  by  this  research. 

When  a  ToC  is  made  explicit  it  is  able  to  be  tested  against  the  realities  of  the  area  of 
operations,  and  revised  as  necessary.  As  Figure  2  illustrates,  if  plans  are  limited  to  listing 
tangible  actions  and  roles,  and  do  not  articulate  the  logic,  assumptions  and  links  between 
them,  (represented  by  the  coloured  portions  of  Figure  2),  then  there  are  a  number  of  critical 
functions  that  cannot  be  effectively  carried  out.  This  may  not  prevent  the  execution  of  a  plan, 
but  it  is  likely  that  it  will  be  carried  out  with  misinterpretations  and  inconsistencies  emerging 
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as  the  plan  is  translated  into  tasks.  By  considering  campaign  plans  through  ToC  lens,  we  are 
now  able  to  identify  a  set  of  critical  campaign  functions  that  will  be  potentially  enhanced  by 
the  illumination  of  a  plan's  logic  and  assumption  linkages.  For  example,  it  is  also  very  difficult 
to  evaluate  the  extent  to  which  a  plan  is  progressing  along  the  path  to  success,  if  that  path  is 
not  described.  Further,  if  a  plan  is  not  delivering  expected  results,  it  is  very  difficult  to 
pinpoint  where  corrections  should  be  made,  or  to  learn  lessons,  if  the  logic  and  assumptions 
that  led  to  error  or  failure  are  not  transparent. 


Figure  2:  Illustration  of  the  necessity  for  articulating  logic  and  assumptions  in  a  plan 


Chapter  5  explores  how  four  of  these  critical  campaign  functions  can  be  enhanced  by  trial 
application  of  a  logic,  assumptions  and  risks  framework. 

3.2.1  Variations  of  Theory  of  Change 

In  military  operations  research,  the  ToC  represents  the  rationale  for  how  operational  activities 
(in  combination  with  other  factors)  will  result  in  desired  outcomes  as  described  in  the 
campaign  objectives  and  end  state.  The  inclusion  of  ToC  in  campaign-level  planning  offers 
transparency  and  clarity  in  the  planning  process,  as  well  as  strong  indicators  of  success  or 
failure,  as  observed  by  the  UK  Ministry  of  Defence's  Development,  Concepts  and  Doctrine 
Centre  (Couzens  2010): 

It  is  only  when  ideas  have  been  put  into  words  that  flaws  in  logic  become  apparent  and  when 
misunderstandings  between  planning  team  members  or  the  planning  staff,  commanders  and  the 
rest  of  the  staff  are  revealed. 
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The  UK  Ministry  of  Defence's  Joint  Doctrine  Note  on  Assessment  (2012)  refers  to  it  as  a  'road¬ 
map'  to  the  desired  condition,  which  should: 

•  Identify  the  overall  aim; 

•  Set  out  the  inputs,  processes  and  outputs  required  to  achieve  that  aim; 

•  Describe  the  logic  that  underpins  the  solution,  explaining  how  and  why  the  proposed 
actions  will  change  a  situation. 

However,  Australia's  Office  of  Aid  Effectiveness  (Dart  2011)  warns  that  a  ToC  must  capture 
more  than  just  inputs  and  outputs,  to  fully  account  for  how  and  why  a  program  will  bring 
about  change.  The  Care  International  UK  (2012)  agrees: 

a  well-articulated  theory  of  change  represents  a  testable  hypothesis  regarding  how  the  planned 
activities  will  contribute  to  achieving  the  desired  results  for  the  programme. .  .Every  action  we  take, 
from  the  overall  goal  of  the  project  to  each  single  activity,  has  a  theory  of  change  behind  it 

Therefore  ToCs  can  be  both  macro  and  micro.  At  the  macro  level  the  ToC  represents  a 
generalised  understanding  of  how  change  occurs  (the  causal  logic)  and  thus  identifies  the 
mechanisms  to  focus  on  in  order  to  achieve  an  overall  effect  (Couzens  2010).  This  has  been 
expressed  in  British  military  campaign  assessment  guidance  (Development  Concepts  and 
Doctrine  CentreDCDC  2012)  as  the  commander's  'big  idea'  of  how  to  bring  the  situationfrom 
the  current  undesired  one  to  the  future  desired  state.  Micro  ToC  can  focus  on  the  changes  a 
program  intends  to  bring  about,  taking  account  of  the  interplay  between  specific  intended 
actions  and  the  program's  context.  Alternatively,  Couzens  differentiates  these  two  levels  as 
top-down  and  bottom-up.  He  refers  to  macro  ToC  applied  to  the  design  and  implementation 
of  a  program,  by  nominating  where  it  is  located  within  the  spectrum  of  strategic  options,  or 
"archetypes  of  change",  from  where  the  lower-level  details  of  how  the  TOC  can  be  derived. 
Micro  ToC,  is  described  as  a  bottom-up  process  of  articulating  how  change  will  occur  by 
linking  actions  to  context  and  consequence  from  the  bottom  upwards. 

Top-Down  Derivation  Bottom-Up  Theories  of  Change 


Figure  3:  A  demonstration  of  how  the  inclusion  of  theories  of  change  which  link  Effects  to  Decisive 
Conditions  can  add  robustness  to  campaign  planning  (Couzens  2010) 
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3.2.2  Potential  for  application 

The  ToC  approach  provides  a  problem  structure  and  theoretical  vocabulary  for  a  form  of 
problem  analysis  that  is  already  occurring  as  part  of  DSTO's  support  to  CA.  To  date  it  has 
been  occurring  as  small,  incomplete  thought  chains  and  analysis  steps  as  part  of  a  largely  ad 
hoc  process.  By  recognising  these  analysis  steps  as  ToC,  we  can  apply  them  in  a  more 
deliberate,  systematic  manner,  cognisant  of  when  they  should  be  commenced  and  ensuring 
that  they  are  carried  out  to  completion,  so  that  full  benefit  is  derived  from  them.  It  is 
concluded  that  the  suitability  of  either  macro  or  micro  ToC  approach  for  our  purposes  is 
dictated  by  the  resources  and  drivers  for  a  program,  and  whether  ToC  is  being  applied  at  the 
planning  stage,  or  later  as  part  of  an  evaluation  phase.  For  example,  if  resource  or  time 
constraints  are  key  drivers,  then  micro  ToC  is  more  likely  to  be  useful,  because  it  is 
consideration  of  what  we  are  able  to  do  that  is  more  likely  to  determine  goals,  and  subsequently 
determine  the  scope  of  the  campaign  objective.  Flowever,  if  the  driver  of  a  program  is  the 
achievement  of  a  priority  objective  that  is  considered  to  be  critical  to  strategic  interests  at  any 
cost,  then  a  macro  ToC  will  be  more  applicable,  starting  with  consideration  of  what  we  wan  t  to 
achieve. 

The  maturity  of  the  plan  will  also  impact  upon  the  choice  of  macro  or  micro  ToC.  For  a  mature 
plan  which  is  already  underway,  planners  are  less  likely  to  be  seeking  to  revise  high-level 
objectives,  but  may  be  willing  to  alter  some  Effects  and  activities;  therefore,  the  top-down 
macro  ToC  is  likely  to  be  more  useful.  Where  ToC  is  being  applied  at  the  initial  planning 
phase,  there  may  be  more  appetite  to  take  a  bottom-up  approach,  where  the  'art  of  the 
possible'  and  consideration  of  the  detail  of  the  operating  environment  determines  the  scope  of 
what  a  program  aspires  to.  Therefore  a  micro  ToC  will  enable  discussion  of  a  program  of 
desirable  activities  and  articulation  of  their  expected  Effects,  the  subsequent  conditions  that 
those  effects  are  assumed  to  create,  and  finally  the  objective  that  will  be  achievable. 

There  are  a  variety  of  applications  for  DSTO's  support  to  ADF  campaign  planning  and 
evaluation,  which  span  the  spectrum  of  phases  described  above,  so  it  can  be  concluded  that 
either  micro  or  macro  ToC  would  be  applicable  at  different  times.  As  a  consequence,  it 
becomes  clear  that  if  ToC  is  to  be  incorporated  into  a  Defence-tailored  framework,  it  must  be 
in  a  form  that  is  adaptable  to  allow  either  macro  or  micro  ToC  to  be  explored.  This  conclusion 
is  echoed  in  program  theory  literature,  below,  which  discusses  the  requirement  for  both 
micro-  and  macro-  program  theories,  and  again  under  different  scenarios  for  framework 
application  in  Chapter  5  below. 

For  Defence's  planning  and  evaluation  purposes,  theory  of  change  is  valuable  for  determining 
and  documenting  what  is  expected  to  be  achieved,  and  goes  part  of  the  way  to  identifying  how 
those  achievements  are  expected  to  come  about.  However,  it  does  not  systematically  provide 
enough  detail  on  how  goals  will  be  achieved,  particularly  in  terms  of  assumptions  about 
intangible  consequences  of  action.  The  absence  of  this  level  of  detail  reduces  the  potential 
comprehensiveness  of  any  set  of  indicators  of  change  derived  from  ToC.  ToC  approaches  also 
do  not  automatically  lead  to  sequencing  of  activities  in  a  form  that  enables  planners  to 
combine  categories  of  effort  or  take  account  of  interdependencies.  Progressing  to  this  level  of 
detail  is  necessary  to  define  specific  triggers  of  when  activities  should  start  or  end  (also 
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referred  to  as  Decisive  Points),  and  to  become  aware  of  contradictions  between  courses  of 
action  and  the  risks  that  need  to  be  monitored  throughout  the  life  of  the  plan. 

3.3  Program  Theory 

Once  a  theory  (or  theories)  of  change  is  established,  the  mechanisms  for  execution  and 
evaluation  need  to  be  established.  Program  theory  is  defined  by  Funnell  and  Rogers  (2011)  as 
an  approach  that  brings  together  ToC  with  a  theory  of  action  to  provide  a  cohesive  plan 
explaining  how,  when  and  why  change  is  intended  to  come  about.  Similar  to  ToC,  the  value  of 
this  theory  does  not  lie  in  creating  something  that  would  not  otherwise  exist  -  a  plan  will 
always  comprise  some  sort  of  intended  change,  and  a  series  of  actions  that  are  anticipated  to 
achieve  that  change.  The  value  of  program  theory  is  the  way  in  which  it  prompts  the 
articulation  and  systematic  consideration  of  intents  and  actions. 

The  formal  history  of  this  theory  stems  from  1960's  America,  when  the  government  began 
investing  huge  sums  in  large-scale  social  development  programs.  There  was  no  precedent  for 
effective  and  transparent  design  of  these  programs,  and  no  ability  to  evaluate  the  extent  to 
which  they  were  delivering  on  intended  outcomes.  Suchman  (1967)  is  attributed  with 
authoring  the  first  guidance  on  program  level  evaluative  research.  This  was  progressed 
during  the  following  decade  by  researchers  such  as  Weiss  (1972)  who  emphasised  the  role  of 
evaluation  concepts  in  program  planning,  and  pointed  to  the  necessity  for  evaluation  practice 
in  the  formulation  of  'program  goals'.  She  also  touched  upon  the  concept  of  complex 
programs  as  systems  that  may  have  'unanticipated  consequences',  and  depicted  program 
components  and  causal  chains  in  flow  charts.  By  the  1980s  the  term  'program  theory'  was  well 
established  with  a  variety  of  definitions,  but  with  the  central  theme  of  a  model  of  how  a 
program  is  supposed  to  work,  to  provide  the  basis  for  implementation  and  evaluation 
(Bickman  1987). 

Development  of  a  program  theory  involves  mapping  out  the  mechanisms  for  change  and 
assumptions,  and  defining  these  as  the  logic  that  underpins  a  program.  A  complete  program 
theory  is  comprised  of  two  elements:  a  theory  of  change  (as  discussed  above)  and  a  theory  of 
action.  Together,  a  theory  of  change  and  a  theory  of  action  will  account  for  how  and  why 
change  is  intended  to  come  about.  The  theory  of  action  is  the  linkages  between  elements  of  the 
logic  of  the  plan,  and  explains  how  action  will  produce  desired  results.  When  articulated  in 
narrative  format,  a  program  theory  comprising  both  a  theory  of  change  and  a  theory  of  action 
can  be  thought  of  as:  "If  x  happens  then  y  will  result  because  x  is  a  precursor  to  y."  Using 
program  theory  enables  greater  inclusion  of  the  context  of  the  operating  environment,  and  the 
effects  caused  by  outside  influences:  "If  x  happens  then  y  will  result  because  x  is  a  precursor  to 
y  so  long  as  p  does  not  occur  in  the  operating  environment." 

The  inclusion  of  program  theory  in  planning  and  evaluation  enables  the  development  of 
indicators  which  will  monitor  the  plan's  logic,  not  just  its  outcomes.  This  means  that  planners 
become  aware  of  when  their  plan  is  not  leading  to  expected  outcomes,  and  its  assumptions 
and  logic  may  need  to  be  reconsidered.  It  is  the  synthesis  of  technological  and  methodological 
expertise  with  subject-matter  expertise  which  characterises  program  theory  and  its  role  in 
planning  and  evaluation  (Williams  and  Morris  2009).  When  used  in  evaluation,  program 
theory  enables  greater  testing  of  whether  a  failure  to  achieve  intended  outcomes  was  due  to 


14 


UNCLASSIFIED 


UNCLASSIFIED 

DSTO-TR-2840 

faulty  theory  of  how  to  achieve  the  outcomes,  or  faulty  implementation  of  the  program  itself. 
Bickman  (1987)  explains  this  in  a  different  way: 

In  the  parlance  of  experimentation,  the  program  is  an  operationalisation  of  the  construct  or 
concept  underlying  the  program.  Conceptualizing  the  program  as  a  complex  set  of 
independent  variables  that  refer  to  a  broader  and  even  more  complex  set  of  variables  (the 
construct)  demonstrates  the  relevance  of  program  theory  to  construct  generalization.  Without 
being  able  to  specify  these  theory  elements,  it  cannot  be  determined  if  a  particular  program  is 
a  valid  operationalisation  of  the  theory  underlying  the  program  [and  whether]...  the 
particular  program  is  a  good  representation  of  the  theory. 

3.3.1  Variations  of  Program  Theory 

When  articulated  in  narrative  format,  a  Program  Theory  comprising  both  a  theory  of  change 
and  a  theory  of  action  can  provide  a  detailed  articulation  of  understanding  of  the  anticipated 
program  from  start  to  finish.  It  can  situate  a  theory  of  action  within  a  defined  program  space, 
with  discussion  of  the  logic  or  theory  of  change,  upon  which  the  program  has  been  defined. 
Further,  a  narrative  program  theory  will  step  through  layers  of  logic  and  corresponding 
action,  giving  opportunity  to  draw  out  how  the  broader  context  within  which  the  program 
occurs,  is  expected  to  impact  on  progress.  These  broader  context  variables  are  referred  to  by 
Funnell  and  Rogers  (2011)  as  "non-program  external  factors".  However,  some  program 
theorists,  such  as  Scheirer  (1987)  draw  a  sharp  distinction  between  program  theory  and 
implementation  theory,  insisting  that  program  theory  should  be  limited  to  an  initial  theory  of 
change,  which  is  later  translated  into,  and  used  to  evaluate,  a  plan  of  action.  This  form  of 
Program  Theory  is  sometimes  referred  to  as  'program  philosophy',  emphasising  that  it 
encompasses  a  "system  of  theories"  which  "comes  before  the  physical  and  behavioural 
manifestations  of  the  program"  (Conrad  and  Miller  1987).  For  the  requirements  of  this 
research,  including  its  immediate  application  to  extant  planning,  it  was  considered  more 
useful  to  apply  a  form  of  Program  Theory  that  does  not  exclude  analysis  of  detailed  structures 
of  action,  giving  the  flexibility  to  consider  either  intent  or  action  as  required.  Similar  to  micro- 
and  macro-  ToC,  discussed  above,  two  tiers  of  program  theory  can  be  defined  as  follows: 

•  Program  micro-theory  -  describes  structural  and  operational  aspects  in  detail  with 
information  on  functions  and  variables.  This  identifies  components  that  can  be 
changed  when  it  is  evident  that  the  program  as  a  whole  cannot  be  changed. 

•  Program  macro-theory  -  describes  the  large-scale  components  and  factors  that 
facilitate  or  constrain  change  within  and  outside  programs  (Shadish  1987). 

Program  Theory  is  also  commonly  expressed  in  various  diagrammatic  forms,  which  are  also 
referred  to  as  logic  models.  As  shown  in  the  example  at  Figure  4  below,  these  are  basically 
variations  on  flow  diagrams  that  visualise  greater  or  lesser  degrees  of  information,  about  the 
linkages  between  the  overall  program  objectives,  and  how  these  are  catergorised  and  broken 
down  into  sub-components.  While  the  degree  of  detail  on  sub-components  in  these  logic 
models  may  vary,  all  logic  models  must  at  least  depict  the  entire  top  level  of  a  program.  The 
form  of  the  model  can  shape  the  points  of  emphasis  for  a  program,  such  as  whether  much  of 
the  high-level  rationale  for  the  program  is  pre-determined,  so  the  focus  of  the  model  is  on 
translating  that  high-level  guidance  into  achievable  goals  and  specific  actions.  Alternatively, 
some  logic  models  provide  more  space  for  illustration  of  the  range  of  activities  that  already 
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exist  or  are  considered  possible,  and  then  allow  for  extrapolation  upwards  to  form  an 
overarching  program. 

The  desired  format  and  points  of  emphasis  of  a  program  theory  model  will  depend  on  its 
context  of  application.  For  example,  as  Bickman  (1987)  observes,  "a  psychologist  may  prefer  to 
develop  a  program  theory  using  individual  cognitions  and  attitudes  as  a  basis,  while  a 
sociologist  may  prefer  to  use  roles  and  organizational  variables.  An  economist  will  probably 
be  most  comfortable  using  a  microeconomic  theory". 


Background  and  theory 
to  task 

(History/  stakeholders 
/scope  of  evaluation  task) 


Program  Logic 
(What  is  the 
Campaign  trying 
to  achieve?) 


Context  of  Program  to 
be  evaluated 

(Considerations, 
constraints  of  Campaign 
and  client  requirements) 


Evaluation 

Questions 

(What  do  we 
want  to  know?) 


Monitoring  Plan 

(What  are  we  doing  and 
what  do  we  need  to  know 
for  accountability?) 


\  Evaluation  Plan 

\  (Consensus  on 
)  frequency,  resources 
/  and  scale) 


Each  comprising: 


Reporting 

Strategy 


\  Implementation 
]  Strategy 


Figure  4:  Example  of  Program  Theory  model  applied  to  an  evaluation  plan  (Hickman  2012) 


A  Program  Theory  should  demonstrate  logical  and  defensible  linkages  between  activities  and 
outcomes.  However,  as  Funnell  and  Rogers  (2011)  point  out,  applying  Program  Theory  and 
creating  a  model  of  linkages,  does  not  guarantee  that  those  linkages  are  robust  or  correct  and 
enduring.  It  is  possible  to  build  a  Program  Theory  model  of  activities  and  outcomes  that  does 
not  reflect  reality,  or  even  likely  capability  or  consequence.  There  are  many  variations  on 
flaws  in  Program  Theory  models,  such  as7: 

•  A  Program  Theory  model  that  comprises  actions  that  are  partial  steps  towards  success, 
but  not  sufficient  to  achieve  an  outcome;  (e.g.  assuming  that  proving  a  target  group 
with  information  about  health  will  be  sufficient  to  achieve  improved  health  outcomes) 


7  Partially  derived  from  Funnell  and  Rogers  (2011),  Pg  43-  52. 
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•  A  comprehensive  Program  Theory  model  that  solves  the  wrong  problem.  This  is  often 
a  result  of  incorrect  assumptions  about  the  causal  factors  for  a  given  situation;  (e.g. 
seeking  to  establish  stability  in  a  conflict-affected  community  by  re-establishing 
governance  and  law  and  order  institutions  that  have  broken  down,  but  failing  to 
address  communal  inter-group  grievances) 

•  A  program  model  that  oversimplifies  causal  chains,  and  is  not  based  upon  adequate 
situation  analysis.  This  is  often  the  result  of  inadequate  effort  devoted  to  establishing 
an  accurate  baseline  of  the  factors  we  are  seeking  to  change  and  often  manifests 
unintended  consequences,  or  lack  of  impact;  (e.g.  an  agricultural  efficiency 
development  program  model  may  require  levels  of  local  participation  and  recourses 
that  are  not  available,  or  may  be  executed,  without  awareness  of  consequences  for 
local  employment  or  social  structures) 

•  Excessive  or  insufficient  consideration  of  intended  outcomes.  Excessive  focus  on 
outcomes  can  lead  to  program  goals  that  are  too  prescriptive,  or  too  low  level,  and  fail 
to  consider  longer  term  or  cumulative  effects.  Insufficient  consideration  of  goals  can 
result  from  a  focus  on  participating  in  a  particular  activity,  and  this  participation,  or 
the  use  certain  techniques  or  assets,  becomes  the  goal.  In  turn,  measures  of  success 
become  lists  of  activities,  or  quantitative  inputs  and  outputs,  rather  than  indicators  of 
the  actual  impact  of  those  activities,  (e.g.  the  success  or  failure  of  a  crime  prevention 
initiative  may  not  be  recognised  if  based  on  prescriptive  targets  of  reduction  in 
particular  crime  statistics  in  a  certain  timeframe.  Similarly,  if  overall  desired  effects  are 
not  well  defined  as  part  of  a  crime  prevention  program  model,  evaluation  may  be 
limited  to  reporting  on  numbers  of  participants  in  focus  groups,  or  percentage  of 
uptake  on  rehabilitation  courses). 

3.3.2  Potential  for  application 

Program  Theory  is  consistent  with  the  intent  of  DSTO's  support  to  operational  planning  and 
evaluation,  which  is  to  render  explicit  the  chain  of  logic  that  underpins  a  program  of  activities, 
so  that  appropriate  actions  and  indicators  of  progress  can  be  identified.  Program  Theory 
provides  a  planning  structure  which  is  ideal  for  this  kind  of  transparency  and  evaluation. 
However,  within  the  Defence  context  of  application,  it  has  a  number  of  limitations  that 
warrant  adaptation. 

Program  Theory  is  most  effective  when  applied  at  the  initial  planning  phase  of  a  program. 
When  applying  it  to  an  existing  plan,  it  may  be  found  that  some  elements  of  the  model  do  not 
exist,  and  the  process  for  retro-fitting  them  may  not  be  desirable  at  a  given  point  in  time.  As 
demonstrated  in  the  examples  of  application  described  in  Chapter  5,  the  research  team  has 
found  that  this  is  often  the  case  in  the  Defence  strategic-level  planning  context,  where  support 
for  planning  or  evaluation  is  requested  after  a  plan  is  well  advanced,  or  even  already  being 
executed.  It  is  more  common  that  support  to  planning  is  requested  during  reviews  and  re¬ 
drafts  of  extant  plans.  In  this  instance  by  converting  the  plan  into  a  Program  Theory  model,  it 
would  be  possible  to  take  a  more  structured,  systematic  approach  by  reviewing  each  of  the 
program  components  and  having  transparency  in  the  impact  of  alterations  of  one  component 
of  the  plan  on  other  components.  It  would  also  prompt  consideration  of  whether  the  plan  is 
subject  to  any  of  the  flaws  that  Program  Theory  warns  against.  However,  the  concept  of 
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converting  an  established  plan  into  a  program  theory  model  is  likely  to  be  rejected  as 
requiring  excessive  time  and  staff  inputs  for  Defence's  very  tight  operational  decision-cycles. 

For  ADF  operational  planning  and  evaluation,  there  is  also  a  pattern  of  reluctance  to  direct 
program  evaluations  or  reviews  at  the  strategic  guidance  level.  Being  a  Government 
department.  Defence's  role  is  to  focus  on  the  ways  in  which  it  can  contribute  to  strategic 
requirements,  rather  than  questioning  or  re-shaping  those  requirements.  This  demands  an 
approach  that  examines  a  program's  logic,  assumptions  and  risks,  like  Program  Theory,  but  in 
a  more  scalable  (ie.  rapid  or  lengthy,  high  or  lower  level)  form. 

3.4  Cognitive  and  Causal  Mapping 

Another  theoretical  approach  which  aims  to  make  explicit  the  intent  and  logic  of  activities  or 
behaviour,  is  cognitive  and  causal  mapping.  The  quality  of  a  program  or  operation  is  directly 
linked  to  the  extent  to  which  all  stakeholders  share  an  understanding  of  its  intent  and  logic 
(explicit  and  implicit).  In  military  terms,  it  has  been  observed  by  Bryant,  Blais  and  Baranski 
(2008)  that  "Ultimately,  shared  implicit  intent  is  the  most  important  aspect  of  command  and 
the  key  determinant  of  success  in  an  operation."  Having  visibility  of  the  mental  model  behind 
a  plan  enables  subordinate  commanders  to  decide  upon  action  that  is  consistent  with  original 
intent,  and  also  enables  assessment  of  whether  there  has  been,  or  is  likely  to  be,  progress 
towards  intended  goals. 

The  term  cognitive  mapping  is  said  to  have  first  been  used  by  Tolman  (1948)  for  a  psychology 
paper  entitled  "Cognitive  Maps  in  Rats  and  Men".  However,  it  was  not  until  the  1980s  that  the 
concept  expanded  from  cognitive  science  into  wider  social  science  and  operations  research 
(Eden  1988).  Cognitive  mapping  is  a  representation  of  a  network  of  how  an  individual  thinks 
about  a  particular  set  of  issues.  Causal  mapping  is  the  same  process  applied  to  a  group's 
thinking.  Most  simply  defined  by  Bryson,  Ackermann,  Eden  and  Finn  (Bryson  2004): 

Maps  allow  exploration  of  issues  and  answers  through  assisting  and 
clarifying  the  content  and  logic  embedded  in  discussion  and  dialogue. 

Indeed,  mapping  prompts  mappers  to  articulate  what  they  think  and 
why. 

It  is  usually  manifest  as  a  very  simple  diagram  with  nodes  representing  relevant  issues  and 
action,  labelled  with  sentences  describing  a  state,  or  activity,  and  arrows  showing  how  one 
action  leads  to  another,  or  how  one  issue  impacts  upon  another.  As  discussed  in  relation  to 
ToC,  and  echoed  as  a  definition  of  cognitive  mapping,  the  value  of  this  method  "is  not  in  the 
tool  itself,  but  the  conducting  of  the  interview  which  will  lead  to  the  establishment  of  the 
cognitive  map"  (Bouyssou  2006). 

3.4.1  Variations  on  Cognitive  Mapping 

Mapping  can  be  used  as  a  pre-decisional  tool;  most  often  through  causal  mapping  as  a  group 
facilitation  technique.  It  can  also  be  used  to  passively  document,  rather  facilitate  an  individual 
or  group  decision  path,  to  provide  transparency  and  detail  to  flesh  out  articulation  of  the 
intent  and  scope  of  a  program.  Finally,  it  may  be  applied  retrospectively,  as  an  attempt  to 
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understand  why  decisions  were  made,  and  to  use  that  understanding  to  choose  appropriate 
courses  of  action,  or  to  determine  where  an  error  may  have  been  made. 

When  applied  to  planning,  cognitive  mapping  can  capture  and  elaborate  on  a  single  line  of 
command  logic,  or  causal  mapping  can  capture  and  compare  the  intent  of  multiple 
stakeholders.  The  utility  of  individual  or  group  mapping  for  drafting  or  evaluating  a  plan  will 
depend  on  the  degree  to  which  a  plan  is  directed  from  a  single  senior  source  or  the  product  of 
multiple  inputs.  Where  a  plan  is  a  top-down  directive,  cognitive  mapping  can  capture  a  senior 
decision-maker's  logic  that  underpins  the  plan,  which  can  then  be  articulated  as  program  logic 
and  be  more  likely  to  be  accurately  interpreted  by  those  executing  the  plan.  This  subordinate 
interpretation  of  program  logic  has  also  been  referred  to  as  Theory  of  Mind  (ToM)  (Vartanian 
2008),  which  is  a  variation  on  cognitive  mapping.  ToM  refers  to  the  formation  of  assumptions 
about  what  another  person  is  thinking  (i.e.  their  cognitive  map).  When  we  do  not  have  direct 
access  to  the  cognitive  map  behind  a  decision  or  plan,  we  can  only  make  assumptions  about 
the  intent  and  expectations  behind  it,  to  inform  how  we  should  behave  in  response.  This  is  a 
common,  informal  mechanism  for  anticipating  and  responding  to  the  behaviour  or 
expectations  of  others,  usually  in  very  immediate  contexts.  For  example,  when  driving  one 
uses  observation  of  the  gaze  of  another  driver,  combined  with  knowledge  of  the  road  ahead, 
to  make  assumptions  (a  ToM)  about  what  they  intend  to  do,  so  that  one  might  be  prepared  to 
respond  if  they  are  slowing  to  turn  a  corner.  In  less  immediate  terms,  when  executing  a  plan, 
it  is  necessary  to  make  assumptions  about  the  cognitive  map  of  planners,  in  order  to  interpret 
how  they  intended  the  plan  to  be  translated  into  action.  Different  stakeholders  may  form  very 
different  ToMs,  depending  on  their  experience  and  access  to  planning  information,  and  this 
will  affect  the  consistency  and  effectiveness  of  overall  execution.  In  military  terms,  the 
accuracy  of  a  subordinate's  ToM  will  depend  on  the  quality  of  strategic  guidance,  their 
experience  and  knowledge,  and  the  degree  of  detail  and  context  provided  in  a  plan.  The 
significance  of  this  is  that  by  using  the  term  ToM,  we  have  a  vocabulary  for  differentiating 
between  the  cognitive  map  (logic)  of  planners  and  what  ToM  (subordinate  assumptions)  are 
made  by  those  who  translate  the  plan  into  action.  This  gives  both  planners  and  evaluators  an 
additional  level  of  granularity  for  assessing  why  a  plan  may  be  contested,  or  be  failing  to 
achieve  its  objectives.  The  plan's  logic  may  be  flawed,  or  perhaps  the  way  in  which  that  logic 
is  being  interpreted  may  be  flawed. 

There  are  a  variety  of  techniques  for  testing  the  extent  to  which  the  cognitive  or  causal  map 
underpinning  a  program  is  understood  or  shared  by  those  responsible  for  executing  it.  These 
include  the  spectrum  of  sense-making  techniques8,  cluster  analysis  or  word  recognition 
software  to  quantify  the  overlap  and  degree  of  similarity  between  mental  models  (Bryant 
2008).  A  range  of  software  tools  have  been  developed  which  can  support  these  forms  of 
analysis  and  comparison.  Flowever,  if  the  aim  of  the  process  is  to  generate  discussion,  or  to 
provide  a  start  point  for  group  engagement,  mapping  can  equally  be  carried  out  using  a 
whiteboard  or  pen  and  paper. 


8  Comprising  a  range  of  group  facilitation,  narrative  and  software  techniques  associated  with  sense¬ 
making  methodology  since  1972.  Dervin,  Dr  B,  Sense-Making  Methodology  Website, 
http:/ / communication.sbs.ohio-state.edu/ sense-making/,  accessed  on  6  November  2012. 
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3.4.2  Potential  for  application 

While  cognitive  and  causal  mapping  align  with  the  DSTO's  aim  to  support  CA  by  drawing 
out  the  logic  and  assumptions  of  programs,  it  requires  engagement  from  decision-makers  to 
articulate  what  their  decision  path  is.  In  the  Defence  operational  planning  context,  gaining 
access  to  high-level  decision  makers,  particularly  during  urgent  crisis  response  planning 
phases,  is  unlikely.  More  frequently,  the  support  provided  to  planning  and  evaluation  is 
occurring  after  key  decisions  have  already  been  made,  and  a  plan  has  already  been  drafted. 
Therefore,  there  is  rarely  an  opportunity  to  apply  mapping  techniques  at  the  point  of  decision 
making.  At  best,  one  may  hope  to  apply  them  retrospectively  or  at  lower  levels  of  planning, 
but  this  is  essentially  an  exercise  in  surfacing  the  assumptions  that  planners  are  making  about 
the  intent  of  decision-makers;  eliciting  their  ToMs.  This  suggests  that  elements  of  ToM  may  be 
the  most  appropriate  form  of  cognitive  mapping  to  apply  to  CA.  Including  questions  from  a 
ToM  perspective  in  an  evaluation  framework  may  elicit  additional  assumptions  and  risks,  as 
it  will  prompt  stakeholders  to  explain  how  they  assume  the  plan  is  intended  to  be  executed, 
and  the  risks  they  associate  with  their  version  of  the  plan. 

Even  where  eliciting  pre-decisional  cognitive  or  causal  maps  is  possible,  they  are  not  sufficient 
in  themselves  to  lead  to  transparent  planning  and  accurate  evaluation.  Causal  mapping  may 
offer  structure  to  a  facilitated  group  discussion,  but  would  need  to  be  used  in  conjunction 
with  a  framework  that  will  draw  out  the  implications  of  the  information  they  reveal.  As  with 
ToC,  it  is  assessed  that  mapping  techniques  may  be  beneficial  as  the  first  step  in  a  framework 
that  proceeds  to  systematically  analyse  the  logic,  assumptions  and  risks  that  emerge  as  part  of 
the  discussion. 

Examining  cognitive  and  causal  maps  may  be  particularly  useful  in  the  identification  of 
indicators  of  progress  for  evaluation.  Our  own  cognitive  map,  or  our  ToM  of  how  others  will 
react  to  a  given  action  is  a  useful  construct  for  reviewing  the  effect  that  operational  tactics  are 
anticipated  to  have  on  a  target  population,  or  on  the  enemy.  For  example  if  our  ToM  on  how 
locals  will  respond  to  foreign  military  presence  is  incorrect,  an  entire  operation  may  be 
undermined.  Alternatively,  if  our  ToM  about  the  extent  to  which  insurgents  will  be  deterred 
by  security  force  presence  is  incorrect,  we  may  find  that  an  operation  is  less  effective  or  takes 
longer  than  expected. 

While  cognitive  and  causal  mapping  does  not  provide  a  single,  scalable  approach  that  is 
applicable  before,  during  and  after  planning,  it  does  provide  a  vocabulary  for  concepts  that 
are  useful  for  development  of  an  alternative  framework.  This  includes: 

•  the  requirement  wherever  possible  to  seek  articulation  of  decision  paths; 

•  comparative  analysis  of  different  cognitive  maps  as  a  mechanism  for  distilling  a 
common  logic; 

•  the  point  at  which  initial  cognitive  maps  begin  to  be  translated  into  ToMs; 

•  the  dual  components  of  a  plan's  logic  that  may  be  made  explicit  -  the  original 
cognitive  map,  and  the  ToM  that  subordinate  decision  makers  and  users  of  the  plan 
construct. 

•  the  difference  between  the  risk  of  flaws  in  original  cognitive  map,  and  risk  of 
inaccurate  translation  into  program  logic  or  action. 
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•  the  ToMs  about  local  population's  response  to  planned  activities  in  different  cultural 
contexts. 

3.5  Synthesis  of  Theories 

This  review  of  relevant  theories  reveals  a  number  of  potential  approaches  to  eliciting  priority 
information  requirements  in  planning  and  evaluation.  Both  ToC  and  Program  Theory  are 
comprised  of  a  number  of  mental  models  rather  than  sets  of  techniques.  In  this  sense  they  are 
theories  rather  than  methodologies.  As  such,  they  provide  guidance  for  scientifically 
considering  a  problem,  but  not  the  actual  techniques  for  applying  that  mental  model 
throughout  the  steps  of  planning  and  evaluation.  Once  a  theory  is  found  to  be  valid,  it 
therefore  requires  a  framework  for  practical  application.  A  large  variety  of  frameworks  and 
techniques  have  been  developed  by  other  agencies  in  the  evaluation  field  which  apply  ToC 
and  Program  Theory9,  but  none  seem  to  meet  the  specific  needs  of  Defence.  Other  agencies' 
adaptations  of  the  theories  provide  partial  solutions,  and  offer  options  for  improving  aspects 
of  the  planning  and  evaluation  processes,  but  each  approach  had  some  characteristics  that 
were  not  suitable  for  the  practical  constraints  outlined  under  2.2  above.  For  example,  many 
existing  program  planning  frameworks  required  a  sizeable,  dedicated  planning  team  to 
conduct  investigations  as  inputs  to  the  process,  which  is  not  currently  within  the  capacity  or 
timeframes  of  ADF  operational  planning. 

What  is  needed  then  seems  to  be  the  development  of  a  bespoke  framework  which  blends  the 
advantages  of  a  variety  of  approaches.  Comparison  of  different  approaches  shows  that  the 
techniques  or  outputs  of  some  are  very  similar,  and  it  is  simply  the  mental  constructs  or  the 
theoretical  origins  that  are  different.  For  example,  making  explicit  one's  own  cognitive  map, 
or  questioning  the  accuracy  of  one's  ToM,  is  essentially  a  cognitive  science  version  of  ToC. 
This  suggests  that  blending  these  approaches  is  possible. 

Consideration  of  campaign  planning  and  CA  using  the  mental  models  offered  by  the  theories 
reviewed  has  yielded  a  number  of  insights  for  the  development  of  a  framework  for 
application.  For  example,  we  are  now  able  to  identify  a  set  of  critical  campaign  functions  (See 
Figure  2)  that  will  be  potentially  enhanced  by  the  illumination  of  a  plan's  logic  and 
assumption  linkages.  These  are: 

1.  Initial  planning, 

2.  Detailed  understanding  of  operational  plan  and  context, 

3.  Evaluation  of  interim  progress, 

4.  Revisions  of  plan, 

5.  Identification  of  indications  of  end  of  military  campaign, 

6.  Lesson  on  success  or  failure. 


9  For  example,  Dr  J.  Dart  et  al,  Evaluation  of  AusAID's  Engagement  with  Civil  Society:  Evaluation  Plan 
(2011)  and  many  other  case  studies  published  by  The  Office  of  Aid  Effectiveness,  AusAID.  Also, 
Enhancing  Program  Performance  with  Logic  Models,  University  of  Wisconsin,  2003, 
http: /  / www.uwex.edu/ ces/lmcourse/  (accessed  7  October  2011).  Many  other  examples  can  be  found 
are  reported  in  the  Evaluation  Journal  of  Australasia,  Australasian  Evaluation  Society. 
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These  six  campaign  functions  provide  a  spectrum  of  operational  functions  that  can  be 
prioritised  for  trial  application  of  a  framework.  Review  of  ToC  has  revealed  that  the  aims  of 
the  current  research  essentially  seek  to  develop  a  way  of  thinking  that  prompts  planners  to  ask 
certain  questions  and  requires  the  articulation  of  answers  in  a  certain  way.  Therefore,  the 
success  of  the  framework  as  a  platform  for  promoting  a  way  of  thinking  becomes  paramount, 
regardless  of  the  exact  techniques  used  to  achieve  it. 

Consideration  of  the  potential  applicability  of  Program  Theory  provides  the  insight  that  the 
concept  of  converting  an  established  campaign  plan  into  a  program  theory  model,  or  any 
other  entirely  new  problem  structure,  is  likely  to  be  rejected  as  requiring  excessive  time  and 
staff  inputs  for  Defence's  very  tight  operational  decision-cycles.  Therefore,  a  theoretical  model 
that  offers  analysis  steps,  through  which  existing  components  of  a  plan  can  be  enhanced  or 
evaluated  in  an  evolutionary  way  is  preferable  to  a  framework  that  involves  revolutionary 
alterations.  By  focusing  on  analysis  steps  for  eliciting  priority  information  requirements,  it  is 
apparent  that  the  categories  of  information  specified  within  Program  Theory  are  of  potential 
value  as  a  checklist  for  analysis  questions.  Similarly,  within  cognitive  and  causal  mapping 
theories,  we  find  that  inclusion  of  analysis  questions  from  a  ToM  perspective  in  an  evaluation 
framework  may  elicit  additional  assumptions  and  risks,  as  it  will  prompt  stakeholders  to 
explain  how  they  assume  the  plan  is  intended  to  be  executed,  and  the  risks  they  associate  with 
their  version  of  the  plan. 

Of  greatest  significance,  all  of  the  theories  reviewed  contribute  to  a  working  vocabulary  for 
discussing  requirements  for  a  credible  and  practical  planning  and  evaluation  process,  and  for 
describing  the  analysis  steps  that  process  contains.  This  vocabulary  can  now  be  applied  in  the 
development  of  a  Defence-tailored  framework. 

3.6  Conclusion 

This  chapter  has  explored  a  number  of  theoretical  platforms  that  offer  mental  models  or 
techniques  that  have  potential  to  enhance  the  DSTO's  support  to  Defence  planning  and 
evaluation.  While  it  would  be  ideal  to  have  additional  time  to  explore  all  potential  theoretical 
sources,  the  insights  derived  from  this  limited  review,  and  the  applicable  analysis  steps 
discovered,  suggest  that  the  essential  theories  have  been  covered.  However,  this  literature 
review  has  also  revealed  the  ways  in  which  each  theoretical  approach  has  limitations  for  our 
research  context,  providing  an  opportunity  to  consider  adaptations  and  blending  of  elements 
of  a  range  of  theories  to  satisfy  Defence's  particular  requirements.  The  next  Chapter  will 
document  the  adaptation  of  these  theories  to  provide  the  building  blocks  for  a  Defence- 
tailored  planning  and  evaluation  tool. 


4  Design  and  Testing  of  the  Logic,  Assumptions  and 

Risks  Framework 


Chapter  3  explored  the  theories  identified  as  being  most  relevant  to  the  aims  of  this  research. 
This  has  produced  a  summary  of  the  most  instructive  and  least  suitable  elements  of  each 
approach  for  use  in  design  of  a  Defence-tailored  framework  to  support  planning  and 
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evaluation.  This  chapter  describes  the  subsequent  design  phase  of  turning  theoretical  insight 
into  a  practical  solution  by  synthesising  beneficial  elements  of  each  approach  into  a  single 
unified  framework.  It  will  then  step  through  each  of  the  elements  of  the  resulting  Logic, 
Assumptions  and  Risks  Framework  (LARF)  framework,  providing  explanation  of  why  and 
how  each  is  included.  Finally  the  method  for  testing  and  assessing  the  LARF  as  a  viable 
solution  will  be  described,  before  this  method  is  put  into  practice  in  the  next  chapter. 

4.1  Design  of  the  Framework 

The  LARF  is  primarily  designed  as  a  platform  for  promoting  a  particular  way  of  thinking.  It 
prompts  analysts  and  planners  to  pose  a  particular  set  of  questions,  which  are  required  to  be 
answered  in  a  particular  way.  In  practice  it  serves  as  a  checklist  for  drawing  out  the  essential 
information  that  planners  and  evaluators  need  to  consider.  The  simplest  way  to  achieve  this  is 
to  create  a  matrix  in  which  a  set  of  priority  information  requirements  is  systematically 
extracted  from  every  aspect  of  a  given  plan  or  program. 

The  construction  of  the  LARF  involved  a  simple  matrix  format  in  a  Word  document,  which 
was  altered  and  exchanged  between  two  DOSC  members  over  several  days.  The  framework 
was  repeatedly  altered  until  it  included  enough  categories  to  elicit  all  of  the  desired 
information,  but  did  not  constrain  responses  or  potential  for  application.  Categories  in  the 
matrix  were  also  considered  with  respect  to  their  scope  for  including  pertinent  evaluation 
questions  identified  in  the  literature  review.  This  resulted  in  the  creation  of  Logic, 
Assumptions  and  Risks  columns,  as  shown  in  Figure  5. 


Line  of  Operation: 

a. 

From 

b. 

To 

c. 

Logic 

d. 

Assumptions 

e. 

Risks 

f. 

Questions 

Effect  1: 

Decisive 

Condition: 

Effect  2: 

Effect  3: 

Effect  4: 

Figure  5:  Components  of  the  LARF  matrix 


With  a  satisfactory  draft  framework  in  place,  an  opportunity  was  created  to  socialise  it  within 
the  DOSC  CA  team,  which  involved  arranging  for  the  LARF  to  become  the  centrepiece  of  an 
internal  workshop,  as  part  of  preparation  for  the  Quarter  4  Operation  Slipper  CA  cycle.  A 
Noting  Brief  was  produced  to  accompany  the  workshop,  which  explained  the  basis, 
mechanics  and  aim  of  the  LARF.  The  internal  workshop  was  held  on  14  December  2011,  and 
while  the  details  of  this  trial  are  covered  under  Section  5.3  below,  it  can  be  noted  here  as  part 
of  the  development  of  the  framework,  that  participant  response  was  entirely  positive  and 
there  was  no  indication  that  any  aspect  of  the  matrix  was  inadequate.  The  design  of  each 
component  of  the  matrix  is  briefly  discussed  below,  but  will  be  discussed  in  detail  under 
"Technique  for  Applying  the  LARF"  in  the  next  chapter. 
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4.1.1  Logic 

The  logic  component  of  the  framework  was  designed  with  a  "To"  and  "From"  column  to 
enable  users  to  define  what  level  of  a  plan,  or  what  aspect  of  strategic  guidance,  they  are 
going  to  nominate  as  a  start  point  by  placing  it  in  the  "From"  column.  This  addresses  two 
potential  challenges  recognised  in  eliciting  logic.  Firstly,  any  form  of  narrative  elicitation  can 
be  time  consuming  and  require  lengthy  transcription,  especially  with  multiple  participants. 
This  would  not  be  suitable  for  many  of  the  anticipated  Defence  applications  of  the  LARF. 
Secondly,  for  a  large-scale,  complex  plan  or  program,  it  can  be  difficult  to  distinguish  between 
multiple  strands  of  logic  or  ToCs,  and  to  examine  each  in  adequate  detail  to  draw  out  a 
comprehensive  set  of  assumptions  or  risks.  For  example,  when  used  for  analysis  of  an  already 
endorsed  Defence  campaign  plan,  users  may  only  be  seeking  to  derive  the  logic  at  the  level  of 
linkages  between  Effects  and  Decisive  Conditions,  because  they  are  not  authorised  to  revise 
the  levels  of  the  plan  above  this.  Participants  are  then  required  to  nominate  what  actions  taken 
in  relation  to  that  aspect  of  the  plan  are  aimed  at  achieving,  by  describing  the  desired  state  in 
the  "To"  column.  This  information  should  be  easily  identifiable  in  any  plan  or  program,  even 
at  the  earliest  drafting  stages,  and  from  here,  the  more  difficult  question  can  be  posed  of  why 
participants  expect  that  column  a)  will  lead  to  column  b)?  It  is  this  explanation  of  causality 
that  can  be  captured  under  the  Logic  column. 

4.1.2  Assumptions 

The  need  to  recognise  and  monitor  assumptions  has  consistently  been  at  the  centre  of  analysis 
in  support  of  Defence  planning  and  evaluation.  Therefore,  a  mechanism  for  systematically 
documenting  them  was  a  necessary  inclusion  in  the  LARF.  Defence,  like  all  other 
organisations,  must  make  plans  in  the  absence  of  certainty.  When  looking  for  the  logic  in  a 
plan,  it  becomes  clear  that  all  plans  are  essentially  sets  of  assumptions.  The  more  complex  a 
plan  is,  the  more  assumptions  it  has  to  make  to  describe  the  current  condition,  the  intended 
actions,  and  the  expected  future  outcome.  The  accuracy  of  a  plan  and  the  effectiveness  of  a 
program  depend  on  the  extent  to  which  the  assumptions  underpinning  it  hold  true  over  time. 
Assumptions  must  be  made  about  the  accuracy  of  available  information,  or  where 
information  is  unavailable,  about  how  a  scenario  is  likely  to  play  out.  Without  assumptions, 
no  policy  or  plans  would  be  possible.  When  assumptions  are  correct,  the  goals  and  actions 
defined  within  a  plan  appear  to  be  logically  coherent  and  achievable.  However,  if 
assumptions  are  not  correct,  then  the  very  foundations  upon  which  the  goals  and  objectives 
we  are  seeking  to  achieve,  and  the  plan  of  action  we  are  pursuing  to  accomplish  them,  may  be 
flawed.  In  DSTO's  provision  of  scientific  expertise,  the  process  of  seeking  to  understand  the 
underlying  assumptions  of  an  operation  can  be  likened  to  evaluating  the  validity  of  a 
hypothesis  (Suchman  1967).  A  plan's  logic  cannot  be  verified  unless  its  underpinning 
assumptions  are  understood  and  assessed  to  be  correct. 

In  the  evaluative  research  field,  assumptions  are  often  classified  into  two  forms  -  value 
assumptions  and  validity  assumptions.  Value  assumptions  relate  to  the  system  of  beliefs  that 
determines  what  is  good,  or  desirable,  for  the  scenario  at  hand.  Validity  assumptions  on  the 
other  hand,  relate  to  anticipated  mechanics  of  bringing  about  change  and  what  will  be 
apparent  as  indications  of  that  change  (Suchman  1967).  This  distinction  is  made  with  various 
labels,  such  as  Chen  and  Rossi's  (1983)  basic  construct  assumptions,  about  human  nature,  the 
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nature  of  the  problem  and  of  the  population,  and  the  boundary,  or  limiting  conditions,  of  the 
effects  of  the  program  -  what  Checkland  and  Poulter's  (2006)  Soft  Systems  Methodology  alerts 
us  to  as  differing  'world  views'.  These  are  contrasted  with  assumptions  about  the  causal 
linkages  about  the  implementation  of  the  program.  Both  of  these  forms  of  assumptions  have 
been  included  in  the  facilitator  questions  suggested  for  populating  the  assumptions  column 
under  the  LARF  technique  in  Chapter  5. 

Without  a  perfect  picture  of  the  present  and  future  environment  in  which  a  plan  is  to  be 
executed,  it  is  inevitable  that  some  initial  assumptions  will  prove  to  be  incorrect.  Most 
commonly,  as  the  environment  evolves,  or  as  more  information  comes  to  light  over  time,  some 
initial  assumptions  are  shown  to  be  flawed.  This  need  not  be  problematic  if  assumptions  are 
explicit  and  monitored.  By  listing  assumptions  in  the  LARF,  either  in  an  initial  workshop,  or 
as  they  become  apparent,  they  can  be  reviewed  as  new  information  becomes  available.  An 
assumptions  column  is  provided  in  the  LARF  linked  to  each  strand  of  logic,  so  that  when  an 
assumption  is  shown  to  be  incorrect,  it  automatically  indicates  that  its  associated  logic 
requires  revision.  In  this  way,  the  LARF  enables  constant  adaptation  of  elements  of  a  plan. 

4.1.3  Risks 

The  risk  component  of  the  LARF  provides  a  space  to  specify  the  'so  what'  of  the  information 
elicited  in  the  other  columns  of  the  matrix.  Each  strand  of  logic  or  assumption  has  a  risk 
associated  with  it,  but  there  are  also  overarching  risks  to  the  achievement  of  each  component 
of  a  plan.  This  column  can  be  populated  with  risks  that  emerge  during  the  course  of 
discussion  of  logic  and  assumptions,  which  may  be  recognised  as  points  of  contention  where 
participants  disagree  about  the  likelihood  or  viability  of  logic  paths.  Capturing  divergent 
views  on  achievability  or  likelihood  as  risks  in  the  LARF  allows  planning  to  proceed  despite  a 
lack  of  consensus. 

Each  risk  should  be  stated  in  terms  of  likelihood  and  impact,  so  that  there  is  an  emerging 
indication  of  which  lines  of  the  plan  are  considered  most  likely  to  fail,  and  which  would  have 
the  greatest  consequence.  This  information  in  the  Risks  column  enables  planners  and 
evaluators  to  make  evidence-based  decisions  on  what  aspects  of  the  plan  should  be 
prioritised. 

4.2  Testing  the  Framework 

To  gauge  whether  the  LARF  is  viable,  it  is  necessary  to  demonstrate  that  it  is  able  to  produce 
the  information  outputs  to  satisfy  the  research  requirements,  and  also  that  it  is  suitable  for  use 
within  the  practical  constraints  of  the  research  context  of  application.  Therefore,  the  LARF  has 
initially  been  tested  using  a  series  of  trial  applications,  which  will  be  detailed  in  the  next 
Chapter.  These  trials  were  conducted  as  part  of  on-going  support  to  operational  tasking  a 
consequence  of  the  operationally-focussed  DOSC  research  environment,  in  which  research 
needs  to  be  immediately  applicable  to  current  operations,  and  where  there  are  rarely 
opportunities  to  test  hypotheses  in  isolated  simulations  or  controlled  trials.  Llowever,  as  will 
be  discussed  below,  the  trials  were  deliberately  selected  to  cover  a  spectrum  of  different 
critical  campaign  functions,  so  that  conclusions  could  be  drawn  about  the  utility  of  the  LARF 
in  a  variety  of  applications.  The  method  for  testing  the  LARF  in  each  of  these  trials  is  by 
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assessing  the  extent  to  which  the  research  aims  set  out  in  Section  1.2,  and  satisfaction  of  the  set 
of  criteria  derived  from  the  practical  constraints  discussed  in  Chapter  2.  These  trials,  aims  and 
criteria  will  be  described  here,  before  the  trial  case  studies  are  presented  and  assessed  in  the 
remaining  chapters. 

4.2.1  Trials 

The  LARF  was  initially  tested  in  a  small  scale  internal  trial  within  DOSC  in  December  2011, 
which  will  be  described  in  detail  in  Section  5.2.2.  This  initial  testing  delivered  some  findings 
on  how  a  LARF  workshop  should  be  facilitated,  and  how  that  facilitation  might  manage  the 
pace  and  content  of  contributions.  Again,  these  findings  are  detailed  under  the  "Application 
and  Results"  chapter  to  follow.  After  the  initial  successful  internal  trial  other  opportunities 
were  sought  to  expand  the  application  of  the  LARF  for  a  variety  of  phases  in  the  planning  or 
evaluation  cycles  other  purposes  within  the  CA  and  planning  cycles.  The  range  of  trials 
selected  cover  four  of  the  six  critical  campaign  functions  identified  under  Section  3.5,  and 
demonstrate  whether  the  framework  is  scalable  and  flexible  enough  to  suit  a  range  of 
applications.  The  four  critical  campaign  functions  upon  which  the  LARF  is  trialled  are: 

1.  Initial  planning, 

2.  Continual  situational  awareness  and  understanding  of  operational  plan, 

3.  Evaluation  of  interim  progress,  and 

4.  Revision  of  a  plan. 

These  functions  are  easily  identified  in  the  high  tempo  tasking  environment,  and  include 
testing  in  Defence  operational  headquarters  (HQJOC),  as  well  as  in  ADF  deployed  areas  of 
operation.  The  findings  from  each  of  these  trials  have  been  integrated  into  the  evolving  LARF 
workshop  approach.  The  second  campaign  function  was  covered  in  the  initial  internal  trial 
(referred  to  in  Section  5.2.2),  which  is  included  in  the  description  of  how  each  trial  proceeded 
is  captured  in  Chapter  5.  There  has  not  yet  been  an  opportunity  to  test  the  LARF  on  the  two 
remaining  critical  campaign  functions  (Identification  of  indications  of  end  of  military 
campaign;  and  Lessons  on  success  or  failure).  These  two  functions  have  become  the  subject  of 
proposed  future  research  (as  will  be  discussed  in  Chapter  7). 

The  LARF  was  trialled  as  part  of  active,  and  also  security  classified  tasks,  so  it  was  not 
possible  to  make  arrangements  for  independent  trial  observers,  or  to  separately  record  process 
observations,  while  simultaneously  facilitating  the  LARF  workshops  and  populating  the 
LARF  matrix.  Therefore,  assessments  of  outcomes  of  LARF  workshops  are  to  date  based  upon 
informal  after-activity  reviews  by  research  team  members,  making  reference  to  the  data 
recorded  in  the  LARF  matrix  during  workshops,  and  the  information  contained  in  CA  MoE 
data  collection  templates10,  in  which  LARF  findings  have  been  incorporated. 

4.2.2  Aims  and  Criteria  that  indicate  a  quality  solution 

A  consistent  theme  of  this  research  is  that  the  viability  of  any  proposed  solution  will  be 
determined  by  the  extent  to  which  it  meets  the  research  aims,  combined  with  its  overall 


10  Both  the  LARF  workshop  matrixes  and  MoE  response  templates  populated  with  operational  data  are 
security  classified  and  so  cannot  be  shown,  or  referred  to  as  specific  examples  in  this  report. 
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practicality.  Therefore,  results  of  LARF  trials  will  be  assessed  against  the  three  research  aims 
defined  in  Section  1.2,  which  are: 

1.  To  highlight  the  priority  information  components  of  a  plan  that  enable  CA. 

2.  To  investigate  whether  it  is  possible  to  develop  a  framework  that  can  be  applied  to  the 
CA  process  to  draw  out  the  information  components  that  have  been  identified  as 
priorities. 

3.  To  consider  whether  the  framework  developed  can  be  applied  to  drafting  of  new 
plans  and  reviews  of  existing  ones,  so  that  priority  information  components  are  clearly 
articulated  in  future  plans. 

In  order  to  achieve  these  aims  within  the  constraints  of  the  research  context,  described  in 
Chapter  2,  research  outcomes  must  meet  a  number  of  practical  criteria.  These  criteria  have 
been  expanded  to  include  insights  as  they  have  emerged  from  trials,  resulting  in  the  following 
six  criteria:  To  be  considered  practical  and  relevant  the  LARF  must: 

1.  have  clear  and  cohesive  outputs; 

2.  be  easy  to  summarise  into  a  variety  of  different  formats  (i.e.  summary  table;  narratives 
for  briefs  or  reports,  key  categories  for  information  requests  or  existing  evaluation 
templates); 

3.  have  a  natural  flow  that  does  not  have  participants  slavishly  stepping  through 
disconnected  components; 

4.  be  flexible  and  scalable  enough  to  allow  extra  time  or  emphasis  to  be  given  to  different 
steps,  in  response  to  participant  preferences; 

5.  be  rapidly  executable  start  to  finish,  without  prior  knowledge  of  underpinning  theory 
or  techniques;  and 

6.  be  executable  with  the  minimum  resource  investment  necessary. 

These  aims  and  criteria  have  shaped  the  development  and  testing  of  the  LARF,  and  will  be  the 
basis  for  assessing  outcomes  in  the  Chapters  to  follow. 

4.3  Conclusion 

The  preceding  discussion  has  set  out  the  research  method  used  to  develop  the  LARF  and  the 
method  by  which  the  viability  and  value  of  the  LARF  can  be  tested.  This  lays  the  foundations 
for  understanding  how  the  LARF  is  constructed  to  integrate  the  information  elements  that 
have  been  consistently  identified  by  the  DSTO  research  team  as  critical  to  effective  planning 
and  evaluation  in  Defence.  This  chapter  has  also  provided  clarity  on  how  research  of  this  type 
may  be  replicated  in  future.  We  will  now  proceed  with  an  explanation  of  the  trials  of  the 
framework  which  can  now  be  understood  to  be  part  of  an  overall  research  method  of 
demonstrating  the  LARF's  immediate  applicability  and  potential  for  future  development. 


5  Application  and  Initial  Results  of  LARF 

The  preceding  chapters  have  captured  the  method  for  the  development  and  testing  of  the 
LARF.  This  chapter  outlines  the  technique  recommended  for  application  of  the  LARF  and  will 
describe  how  the  framework  has  been  tested  through  application  to  three  different  Defence 
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operations  in  four  different  critical  campaign  functions11.  The  LARF  has  proven  to  be  an 
effective  tool  in  each  instance,  with  a  number  of  unexpected  advantages.  The  detail  of  the 
three  operations  to  which  the  LARF  has  been  applied  cannot  be  discussed  in  an  Unclassified 
format.  However,  a  brief  definition  of  each  of  the  operations  is  given  in  Section  5.2  below  to 
enhance  understanding  of  the  terminology  and  context,  before  the  application  and  the  results 
are  discussed  in  generic  terms. 

5.1  Technique  for  applying  the  LARF 

A  structured  discussion  format  approach  has  been  used  as  the  means  for  drawing  out  the 
information  and  populating  the  LARF.  Leveraging  off  the  techniques  commonly  used  in 
Program  Theory,  ToC,  and  Cognitive  Mapping  guided,  DSTO  analysts  facilitating  a 
structured  discussion  process,  introducing  participants  to  the  LARF  concept  before  leading 
them  through  each  of  the  matrix  categories.  This  format  provides  a  significant  degree  of  the 
adaptability  the  study  is  seeking,  because  facilitators  can  use  the  LARF  matrix  to  prompt 
questions  and  drive  discussion  where  information  does  not  flow  freely  or  where  participants 
are  unsure  of  what  it  required.  Alternatively,  facilitators  may  use  the  LARF  matrix  as  a 
reference  point  to  anchor  discussion,  where  information  flows  freely  from  participants,  but 
needs  to  be  focused.  In  this  instance  facilitators  can  clarify  whether  issues  that  emerge  are  best 
captured  as  Logic,  Risks  or  Assumptions.  The  LARF  was  designed  so  that  each  of  the 
categories  does  not  have  to  be  worked  through  in  sequence,  and  in  some  instances,  it  may  be 
easiest  to  go  straight  from  Logic  to  Risks,  and  the  rest  flows  from  there.  The  focus  at  all  times 
is  on  prompting  useful  insight,  rather  than  slavishly  populating  the  template.  However,  if  it  is 
too  difficult  to  think  of  content  for  a  particular  section  of  the  template,  this  may  raise 
questions  about  the  validity  or  appropriateness  of  that  component  of  the  plan. 

The  LARF  is  also  designed  to  enable  the  flexibility  to  select  the  appropriate  scale  for  the  a 
given  task.  A  facilitator  is  able  to  initially  establish  what  the  LARF  application  is  aimed  to 
achieve,  and  at  what  level  are  component  parts  of  the  plan  or  program  changeable.  This  will 
determine  what  level  of  change  components  are  placed  in  columns  a)  and  b),  which  in  turn 
will  shape  the  level  at  which  logic  is  drawn  out  for  analysis  in  column  c). 

5.1.1  Logic 

There  are  a  number  of  considerations  for  LARF  facilitators  when  populating  this  component 
of  the  matrix.  Firstly,  the  aim  of  the  logic  column  is  to  state  how  column  a)  is  expected  to  lead 
to  column  b)  in  the  simplest  possible  terms.  Restating  the  aims  or  objectives  as  set  out  in  the 
plan  or  strategic  guidance  will  have  little  value.  Instead,  participants  should  be  encouraged  to 
clearly  interpret  the  ToC  logic  from  a)  to  b)  as  they  understand  it,  so  that  the  veracity  of  the 
logic  can  be  seen  in  the  absence  of  euphemisms,  political  semantics  and  organisational 
terminology.  For  example,  logic  statements  like  "Disruption  of  insurgent  group  X's  supply  of 
component  Y,  through  consistent  presence  on  transport  routes  will  degrade  their  capability  to 
execute  deliberate  operations  in  the  area",  can  be  simply  stated  as  "If  access  to  resources  is 
made  difficult,  the  bad  guys  won't  be  able  to  attack".  Stated  this  basically,  inherent 
assumptions  and  risks  are  more  obvious  without  the  distraction  of  detail  and  context-specific 


11  As  defined  under  Sections  3.5  and  4.2.1. 
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terminology.  This  facilitator's  strategy  of  translating  narratives  into  simplified  statements  and 
echoing  them  back  to  participants  may  also  overcome  hesitation  by  participants  to  verbalise 
logic  narratives.  Reluctance  to  share  knowledge  has  been  observed  on  occasion  in  trials,  and 
sometimes  attributed  to  concern  by  a  participant  that  their  level  of  knowledge  and  language 
may  not  be  understood  by  the  rest  of  the  workshop  group.  If  this  is  suspected,  facilitators  may 
ask  the  participant  to  offer  a  narrative  of  logic  in  their  own  words,  as  they  would  if  everyone 
in  the  room  was  an  expert12.  The  facilitator  can  then  encourage  the  group  to  interpret  the 
narrative  and  echo  it  back  to  the  participant  in  simple  terms,  until  there  is  mutual  satisfaction 
that  the  logic  has  been  captured. 

Secondly,  drawing  on  the  theoretical  insights  from  causal  and  cognitive  mapping,  LARF 
facilitators  are  encouraged  to  be  mindful  of  the  two  types  of  logic  that  may  be  made  explicit  - 
the  original  cognitive  map,  and  the  ToM  that  subordinate  decisionmakers  and  users  of  a  plan 
construct.  When  a  logic  narrative  is  elicited  through  the  approach  described  above,  it  may  be 
useful  to  ask  whether  participants  consider  this  to  be  reflecting  their  own  cognitive  or  group 
causal  map,  or  whether  it  is  their  interpretation  of  logic  drawn  from  elsewhere.  By 
acknowledging  that  our  logic  may  be  simply  an  interpretation  of  guidance  from  another 
source  or  a  ToM  about  senior  command  intent,  we  can  highlight  and  question  assumptions 
about  intent,  the  credibility  of  the  source  of  that  logic,  and  the  risk  of  our  interpretation  being 
accurate. 

The  LARF  does  not  assume  that  participants  can  simply  be  asked  to  articulate  the  logic  behind 
their  plan  to  begin  populating  the  matrix.  As  cognitive  mapping  has  shown,  people  may 
frequently  be  able  to  state  their  goals,  or  describe  an  activity  they  wish  to  undertake,  but 
articulating  the  thought  process  behind  why  they  seek  those  goals,  or  why  they  believe  one 
course  of  action  is  preferable  to  another,  is  more  challenging.  Therefore,  trials  have  shown  that 
when  attempting  to  encourage  planners  or  evaluators  to  articulate  these  mental  constructs,  it 
is  often  useful  to  ask  for  a  narrative  of  what  they  saw  as  their  start  point,  and  the  story  of  how 
they  envisioned  progressing  from  there  to  the  desired  end  state.  This  is  aligned  with  the 
technique  for  surfacing  logic  through  ToC  or  cognitive  mapping. 

It  may  also  be  useful  for  facilitators  to  consider  whether  the  logic  strands  that  are  emerging 
are  macro  or  micro  ToCs.  This  may  provide  a  reference  point  for  facilitators  to  identify  what 
level  of  logic  is  sought,  and  where  necessary  to  prompt  participants  to  draw  out  more  micro 
or  macro  ToC  logic.  For  example,  if  the  LARF  is  being  applied  at  the  early  stages  of  plan 
drafting,  or  as  part  of  an  entire  plan  review,  it  will  be  useful  for  the  logic  component  to 
address  the  'big  picture'  questions.  Surfacing  these  macro  ToCs  allows  revision  and  validation 
of  the  entire  premise  of  a  plan,  as  well  as  the  assumptions  and  risks  associated  with  it.  These 
macro-level  ToC  questions  may  include: 


12  This  technique  is  derived  from  Shawn  Callahan's  article  in  "Anecdotally"  Newsletter  January  2013,  Anecdote  Pty 
Ltd.,  a  commercial  consultancy  specialising  in  narrative  technique  for  business  storytelling,  strategy  and  employee 
engagement,  Melbourne. 
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•  What  is  our  organisation's/ section's  mandate  as  part  of  the  whole  agency/ national 
effort? 

•  What  are  the  thresholds  for  success/ failure  -  what  degrees  of  each  are  acceptable? 

•  What  interdependencies  and  flow-on  effects  are  we  taking  account  of? 

•  What  longer-term  outcomes  do  we  seek,  and  what  interim  outcomes  and  contextual 
conditions  are  necessary  and  sufficient  to  produce  those  longer  term  outcomes? 

•  How  will  our  intervention  be  received  and  perceived  by  locals? 

•  Are  we  seeking  sustainable  change? 

•  What  aspects  of  the  program  are,  and  what  are  not,  at  our  discretion? 

•  Who  are  the  stakeholders  in  this  initiative?  What  do  we  need  them  to  do,  and  what  are 
we  assuming  they're  doing?  When  do  they  need  to  do  it?  What  are  the  links  between 
different  stakeholders? 

•  What  will  be  happening  in  the  surrounding  environment  while  we  are  achieving  the 
desired  changes? 

Also,  where  the  LARF  is  applied  retrospectively  to  an  existing  plan,  macro  ToC  logic  can  be 
used  as  a  form  of  'gap  analysis'  to  highlight  divergences  between  stated  objectives  and 
planned  actions;  ie.  between  the  macro  ToC  and  events  in  reality  to  date.  Evident  divergences, 
may  then  be  noted  as  flawed  assumptions  or  risks.  By  applying  this  approach  to  the 
remainder  of  the  program  or  plan  at  the  same  nominated  level,  there  may  be  some  repetition 
of  logic  for  various  aspects.  Where  there  is  repetition  of  logic,  this  can  be  noted  as  an 
assumption  or  risk  of  interdependency  between  two  aspects. 

Alternatively,  if  the  aim  of  the  planning  or  evaluation  task  that  the  LARF  is  being  applied  to  is 
to  examine  the  actions  or  chains  of  events  that  will  achieve  higher  level  objectives,  then  a  focus 
on  micro-level  ToC  logic  will  be  more  useful.  A  LARF  facilitator  may  identify  that  participants 
are  focusing  on  re-stating  the  macro-level  logic  contained  strategic-level  guidance,  whilst 
trying  to  review  the  lower  level  activities  or  Effects  of  a  plan.  This  may  necessitate  re-focus  on 
micro  ToCs  to  reveal  the  logic  of  sequencing  and  causality  of  activities,  and  explain  how  a 
given  action  is  expected  to  enable  subsequent  actions,  or  to  lead  to  a  given  change.  Some  of 
the  aspects  of  logic  that  we  would  be  seeking  to  draw  out  in  micro  ToCs  are: 

•  Statements  on  how  targets  are  expected  to  respond  to  intended  actions; 

•  Any  critical  interdependencies  between  stakeholders  or  actions; 

•  Capabilities  required  of  our  own  force  and  others; 

•  Triggers  for  initiation  and  expected  duration  for  actions; 

•  Visibility  of  impact  and  key  indicators  of  end  of  activity. 

When  using  the  LARF  to  draw  out  or  revise  evaluation  frameworks,  micro  ToC  logic  is  more 
likely  to  describe  intended  change  in  sufficient  detail  to  reveal  indicators  of  progress  and  the 
assumptions  and  risks  associated  with  them.  The  research  team  has  found  in  previous 
experience  that,  if  devising  or  validating  evaluation  metrics  for  a  particular  portion  of  a 
program  is  difficult,  it  is  often  because  the  logic  behind  that  part  of  the  program  is  not  well 
understood.  Application  of  LARF  with  a  lower-level  focus  can  assist  in  drawing  out  that  logic 
and  the  indicators  of  progress  associated  with  it. 
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5.1.2  Assumptions 

During  a  LARF  workshop,  assumptions  are  likely  to  emerge  as  part  of  broader  discussion, 
and  can  be  noted  by  the  facilitator  in  the  template  as  they  arise.  When  populating  the  LARF, 
facilitators  should  be  seeking  to  prompt  surfacing  of  the  following  forms  of  assumptions: 

•  definition  of  the  nature  of  the  issue  for  resolution; 

•  assertions  about  connections  between  long-term,  intermediate  and  early  outcomes; 

•  expectations  that  inputs  and  mechanisms  for  required  action  are  available. 

•  belief  that  all  pre-conditions  have  been  identified  and  are  in  place; 

•  statements  anticipating  certainty  of  links  between  activities  and  extent  of  impact; 
value  judgements  about  standards  and  acceptable  behaviours; 

•  contextual/ environmental  factors  that  support  or  hinder  progress  towards 
outcomes.13 

Several  of  these  types  of  assumptions  relate  to  expectations  of  the  way  that  other  people  think, 
and  in  this  respect,  are  aligned  with  the  ToM  defined  in  Chapter  3  under  cognitive/ causal 
mapping  theory.  The  significance  of  this  is  that  it  reminds  facilitators,  when  prompting 
examination  of  the  framework's  logic  column,  to  ask  what  are  the  assumptions  contained 
within  it  about  how  others  will  perceive,  and  behave  in  response  to,  a  program  of  activity? 
Are  these  culturally,  geographically  or  time  dependent? 

It  is  likely  that  a  set  of  assumptions  will  be  documented  for  each  strand  of  logic.  However,  it  is 
also  possible  that  participants  will  sometimes  be  able  to  identify  their  assumptions  about  a 
component  of  a  plan  before  they  are  able  to  define  the  logic  that  those  assumptions  stem  from. 

5.1.3  Risks 

The  risks  listed  in  the  LARF  prompt  participants  to  consider  the  consequences  or  likelihood  of 
failure  associated  with  each  component  of  the  plan.  Risks  may  emerge  in  two  forms:  the  risk 
of  each  assumption  being  flawed;  and  the  risks  associated  with  failure  to  achieve  each  element 
of  the  plan.  Both  can  be  listed  in  the  Risks  column,  and  both  require  participants  to  specify 
likelihood  and  anticipated  impact  of  each  risk.  Where  participants  struggle  to  decipher  the 
logic  or  assumptions  for  a  given  component  of  a  plan,  facilitators  may  find  that  discussion  can 
be  prompted  by  starting  with  identification  of  the  risks  associated  with  a  given  component  of 
a  plan.  While  it  is  obviously  not  constructive  to  record  each  and  every  possible  risk,  users  of 
the  LARF  may  make  assessments  on  those  risks  that  are  significant  enough  to  be  noted  in  the 
LARF.  It  is  advisable  that  the  criteria  for  significance  of  risk  is  established  prior  to 
commencement  of  LARF  workshops,  (i.e.  degrees  of  likelihood  and  impact,  or  degree  of 
consensus)  so  that  time  is  not  wasted  documenting  minor  and  unlikely  risks,  and  also  so  that 
significant  risks  can  be  documented,  even  if  there  is  not  consensus  on  their  likelihood.  There 
are  some  risks  that  decision  makers  are  prepared  to  take,  because  their  impact  or  likelihood  is 
low  in  comparison  to  the  perceived  benefits.  Other  risks  may  not  have  been  considered,  but 
need  to  be  noted  for  monitoring,  mitigation  or  further  investigation  before  a  plan  can  proceed. 


13  Partially  drawn  from  A.  Anderson,  Theory  of  Change  as  a  Tool  for  Strategic  Planning:  A  report  on  early 
experiences,  The  Aspen  Institute  Roundtable  on  Community  Change,  2004. 
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Facilitators  are  encouraged  to  ask  participants  to  consider  the  risk  that  some  of  the  key  terms 
used  in  logic  or  planned  objectives,  could  be  defined  differently  by  different  stakeholders. 
This  is  derived  from  a  research  and  planning  technique  known  as  "the  believing  game" 
(Elbow  1973)  where  participants  are  asked  to  place  themselves  in  the  position  of  believers  in  a 
given  assertion  or  process,  to  surface  all  of  the  perspectives  on  how  it  may  be  correct.  This  can 
be  coupled  with  "the  doubting  game",  where  opposing  perspectives  are  drawn  out.  Peter 
Elbow,  who  is  attributed  with  introducing  these  techniques  suggests  that  the  believing  game 
allows  you  to  project  yourself  into  another's  point  of  view,  to  try  it  on  for  size,  to  "try  to  have 
that  experience  of  meaning".  For  the  purposes  of  LARF,  consideration  of  alternative 
definitions  may  highlight  the  risk  of  misinterpretation,  or  lack  of  clarity  in  a  plan. 

5.2  Trial  Applications 

The  technique  described  above  has  been  consistently  used  for  applying  the  LARF  to  all  trials 
to  date.  As  stated  previously,  the  details  of  these  ADF  campaigns  and  the  specific  outcomes  of 
the  LARF  trials  are  security  Classified.  However,  each  campaign  is  briefly  defined  below  to 
provide  context  for  the  trials,  which  are  described  in  detail  under  sections  5.2.1  -  5.2.4 

Operation  SLIPPER 

Australia's  military  contribution  to  the  International  Security  Assistance  Force  (ISAF)  in 
Afghanistan  is  deployed  under  Operation  SLIPPER.  Australia's  military  contribution  includes 
an  annual  average  of  1,550  Australian  Defence  Force  personnel  deployed  within  Afghanistan. 
These  numbers  vary  depending  on  operational  requirements  and  shifting  seasonal  conditions. 
Approximately  800  personnel  provide  support  from  locations  within  the  broader  Middle  East 
Area  of  Operations,  including  our  maritime  commitment.  In  keeping  with  the  ISAF  strategy  to 
strengthen  civilian  engagement  in  Afghanistan  and  to  better  integrate  civilian  and  military 
efforts,  Australia  now  has  around  50  civilians  working  in  Afghanistan,  in  addition  to  around 
ten  Defence  civilians.  Australia's  substantial  military,  civilian  and  development  assistance 
focuses  on: 

•  Training  and  mentoring  the  Afghan  National  Army  4th  Brigade  in  Uruzgan  province 
to  allow  transition  of  lead  security  responsibility  for  the  province. 

•  Building  the  capacity  of  the  Afghan  National  Police  to  assist  with  civil  policing 
functions  in  Uruzgan; 

•  Helping  improve  the  Afghan  Government's  capacity  to  deliver  core  services  and 
generate  economic  opportunities  for  its  people;  and 

•  Operations  to  disrupt  insurgent  operations  and  supply  routes  utilising  the  Special 
Operations  Task  Group.14 

Operation  ASTUTE 

Operation  ASTUTE  is  the  ADF's  continuing  contribution  to  the  maintenance  of  peace  and 
stability  in  East  Timor,  following  a  request  from  the  Government  of  Timor-Leste  to  the 
Australian  Government  in  2006.  The  Australian-led  International  Stabilisation  Force's  role  in 


14  Australian  Department  of  Defence  public  website, 

http:/ /www.defence.gov.au/ op/ afghanistan/info/ general.htm,  accessed  on  13  February  2013. 
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Timor-Leste  has  now  changed  to  the  redeployment  of  people  and  equipment  following  the 
termination  of  the  United  Nations  Technical  Arrangement.  The  current  agreement  with  the 
Government  of  Timor-Leste  will  see  International  Stabilisation  Force  troops  start  to  depart 
Timor-Leste  in  late  November  and  early  December  and  current  bases  remediated  for 
handover  by  mid  -  2013. 15 

Regional  Campaign  Plan 

In  recent  years,  the  ADF  has  developed  a  Regional  Campaign  Plan  to  define  and  coordinate 
ADF  activities  and  resources  devoted  to  all  forms  of  engagement  across  the  Asia-Pacific 
region.16 

The  LARF  has  been  trialled  on  different  critical  functions  in  each  of  these  campaigns  as 
follows: 

•  an  initial  planning  tool  -  to  test  validity  of  proposed  objectives,  draft  underlying 
detail  of  those  objectives,  and  concept  of  how  their  achievement  will  be  assessed 
(Regional  Campaign  Plan); 

•  a  group  learning  exercise  for  building  situational  awareness  -  the  basis  of  a  forum 
for  exchange  of  ideas  and  knowledge,  for  experienced  and  inexperienced.  Also 
served  as  an  effective  team-building  exercise  which  revealed  individual's  areas  of 
expertise,  strengths  and  weaknesses  (Operation  Slipper); 

•  a  structured  format  for  revision  and  validation  of  an  existing  plan  and  evaluation 
framework  (Operation  Astute); 

•  a  tool  for  generation  of  a  set  of  risks  and  assumptions  to  be  added  as  indicators  to 
an  existing  set  of  measures  of  effectiveness  (Operation  Astute). 

5.2.1  Initial  Planning 

The  application  of  the  LARF  in  early  phase  campaign  planning  was  primarily  aimed  at 
providing  structure  for  the  team  to  draw  out  the  information  they  had  identified  as  being 
critical  for  inclusion  in  a  quality  plan.  This  was  in  recognition  of  the  fact  that  the  longevity  and 
scalable  flexibility  of  a  plan  depends  on  whether  it  is  written  in  a  way  that  articulates 
sufficient  intent,  logic  and  explanation  of  what  combinations  of  activity  are  expected  to  lead  to 
progress  and  ultimately  success.  The  LARF  was  developed  to  reveal  the  logic  behind  why  a 
given  set  of  activities  (i.e.  a  set  of  military  campaign  Effects)  is  assumed  to  lead  from  the 
undesirable  state  to  the  desired  one  ( Decisive  Condition  or  Endstate).  Provision  of  structure 
during  this  phase  was  considered  necessary  to  demonstrate  rigorous,  comprehensive 
consideration,  and  also  to  provide  momentum  and  clarity  for  planners  trying  to  navigate  their 
way  through  the  synthesis  of  a  broad  number  of  complex  variables  into  a  single  cohesive  plan. 

The  LARF  was  first  used  for  this  purpose  during  the  drafting  phases  of  an  ADF  Regional 
Campaign  Plan  for  engagement  across  the  entire  Asia-Pacific  region  from  mid  2011.  When  the 
DSTO's  support  was  requested,  the  draft  plan  already  had  proposed  objectives,  based  on 


15  Australian  Department  of  Defence  public  website, 

http:/ /www.defence.gov.au/op/eastTimor/index.htm,  accessed  on  13  February  2013. 

16  HQJOC  J5  Plans  Branch  email  to  Alison  Hickman,  14  November  2012. 
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strategic  guidance,  but  the  task  of  confirming  that  guidance  had  been  comprehensively 
represented  remained,  as  well  as  teasing  out  the  underlying  detail  of  those  objectives,  to  scope 
of  how  they  are  to  be  achieved  and  how  success  or  failure  would  be  assessed.  Therefore,  the 
LARF  was  used  as  the  basis  of  a  workshop  to  review  each  of  the  draft  objectives  and  consider 
how  the  causes,  opportunities,  consequences,  stakeholders,  and  desired  changes  in  each 
objective  were  being  translated  into  plan  logic.  Flere  the  LARF  would  provide  structure  to 
what  was  essentially  a  problem  definition  and  scoping  exercise,  to  prompt  discussion  of  the 
planners'  ToC,  and  generate  a  shared  concept  of  scope  for  the  program,  and  where  it  sits 
relative  to  the  activities  of  partner  agencies  and  the  broader  operating  environment.  It  was 
determined  at  the  outset  that  the  focus  of  the  LARF  for  this  purpose  would  be  at  the  macro¬ 
theory  level,  given  that  the  aim  at  this  stage  was  to  analyse  the  higher  level  logic  of  the  plan. 
Llowever,  when  assessing  the  risks  and  assumptions  of  this  higher-level  logic,  the  emergence 
of  any  micro-theories  was  also  considered  valuable  for  understanding  what  would  be 
required  to  manifest  change  and  when. 

In  order  to  get  the  best  possible  insight  into  each  aspect  of  the  plan,  representatives  from  each 
relevant  capability  component  of  the  ADF's  L1QJOC  were  invited  to  participate.  Given  that  the 
LARF  had  been  seen  in  other  applications  to  give  new  understanding  even  to  those  very 
familiar  with  the  plan  or  subject  matter,  the  workshop  also  served  as  a  consultation  forum, 
enhancing  the  collective  understanding  of  the  intent  and  implications  of  the  plan.  In  future, 
representatives  from  other  Government  agencies  could  also  be  included. 

The  workshop  was  run  for  half  a  day  with  each  participant  having  received  a  preparation 
pack  of  the  draft  plan  and  a  two-page  noting  brief  explaining  DSTO's  role  in  support  of  the 
task,  the  structure  and  process  of  the  LARF,  and  a  list  of  immediate  and  subsequent  outcomes 
for  the  workshop  that  had  been  agreed  with  the  planners  (Hickman  2012a).  These  were: 

Immediate 

•  A  LARF  that  documents  comprehensive  consideration  about  the  goals  and 
implications  of  each  aspect  of  the  plan.  Ideally  this  content  will  represent  the  consensus 
of  the  group,  but  possibly  not  on  every  detail. 

•  An  enhanced  understanding  by  all  participants  of  the  intent,  vision  for  execution, 
interdependencies  and  information  requirements  throughout  the  life  of  the  plan. 

Subsequent 

•  A  detailed  description  of  each  of  the  plan's  Effects,  which  can  be  inserted  into  the  draft 
Campaign  Plan,  providing  the  kind  of  guidance  that  will  enable  future  operational  and 
tactical-level  planning,  monitoring  and  review. 

•  Visibility  of  any  primary  risks  that  are  common  across  multiple  Effects,  which  can  be 
highlighted  and  prioritised  for  monitoring  or  prevention. 

•  A  set  of  assumptions  and  risks  that  can  be  used  in  the  development  of  an  evaluation 
framework  with  Measures  of  Effect  that  are  directly  related  to  the  logic  behind  each 
Effect. 

An  electronic  version  of  the  LARF  matrix  was  projected  onto  a  large  screen,  which  had  the 
draft  plan's  objectives  listed  in  the  "To"  column,  and  the  draft  Effects  in  the  "From"  column. 
One  member  of  the  DSTO  team  acted  as  facilitator,  and  a  second  was  responsible  for 
populating  the  matrix  as  discussion  progressed. 
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From  the  outset,  participants  were  encouraged  to  step  away  from  repeating  explanations  they 
were  familiar  with  from  draft  documents,  and  to  state  the  logic  in  their  own  most  simple 
terms  of  how  the  Effects  in  column  a)  were  expected  to  lead  to  the  achievement  of  the 
Objectives  in  column  b).  These  simple  statements  revealed  divergences  in  interpretation  of 
intent,  and  expectation  of  mechanisms  for  achieving  it.  Where  these  divergences  arose 
amongst  participants,  they  could  be  noted  as  assumptions.  Discussion  of  logic  eventually 
brought  about  a  consensus  with  some  participants  being  able  to  justify  their  version  of  logic 
as  being  derived  from  senior  directives,  or  relevant  experience.  On  other  occasions, 
participants  were  willing  to  agree  to  a  given  statement  of  logic,  as  long  as  a  number  of  risks 
associated  with  it  were  noted  in  the  LARF,  and  the  draft  plan,  for  on-going  monitoring. 
Deriving  the  logic  of  some  Effects  took  longer  than  others,  and  the  viability  of  those  that  were 
particularly  problematic,  or  rested  on  excessive  assumptions  or  risks,  were  questioned.  As  a 
result,  a  number  of  Effects  were  re-drafted,  or  moved  to  contribute  to  the  achievement  of  a 
different  Objective.  Duplications  in  logic,  or  the  apparent  risk  of  high  degrees  of 
interdependency,  also  highlighted  the  need  to  merge  or  delete  overlapping  Effects.  A  broader 
understanding  of  the  operational  context  was  achieved,  including  recognition  of 
opportunities  and  risks  related  to  other  Government  agency  involvement  in  the  region. 

At  the  completion  of  the  workshop  and  subsequent  tidying  up  of  the  resulting  LARF  matrix, 
all  of  the  intended  outcomes  had  been  achieved.  This  was  manifest  in  a  matrix  populated 
with  revised  Effects  and  grouping  of  Objectives,  detail  on  the  logic,  assumptions  and  risks  for 
each  component  of  the  plan,  and  narrative  summaries  of  the  overarching  threads  of  logic,  and 
the  primary  assumptions  and  risks  of  the  plan.  This  was  provided  to  planners  for  integration 
into  the  draft,  and  was  also  held  by  the  DSTO  team  to  form  the  basis  of  identification  of 
metrics  and  indicators  for  an  evaluation  framework  (an  example  of  which  is  discussed  in 
detail  at  5.2.4  below). 

5.2.2  Continual  Situational  Awareness 

Any  form  of  program  or  strategic-level  planning  and  evaluation  is  a  large  and  complex  task 
that  necessitates  team  input,  from  individuals  with  different  expertise.  Whether  a  new  plan  is 
being  created,  or  an  existing  one  is  being  revised  or  evaluated,  establishing  and  maintaining 
sufficient  knowledge  within  a  team  is  a  challenge.  Reviews  of  ADF  Campaign  Planning  have 
observed  that  the  quality  of  campaign  design  is  dependent  upon:  understanding  of  the 
situation;  the  quality  of  thinking  and  judgement  that  produces  ideas  to  exploit  the  potential 
for  change;  and,  the  quality  of  processes  that  challenge  conjectures  (Smith  2011).  Sometimes 
added  to  this,  is  the  challenge  of  establishing  a  team  across  organisations,  where  the  extent  or 
concept  of  suitable  knowledge  or  situational  awareness,  is  not  understood.  Through  their 
experience  in  supporting  both  planning  and  evaluation  in  Defence,  the  DSTO  research  team 
has  found  that  base-lining,  developing  and  sharing  knowledge,  is  a  valuable  step  at  the 
initiation  of  a  task. 

It  can  usually  be  assumed  that  time  will  be  limited,  so  some  form  of  rapid  group  learning 
exercise  was  considered  desirable  to  establish  shared  concepts  and  focus  team  members' 
thinking  at  the  initiation  of  a  task.  The  investment  of  one  day  for  this  form  of  development 
and  consolidation  activity  is  considered  to  be  a  valuable  investment  in  the  ability  of  the  team 
to  communicate,  assign  roles  and  prioritise  effort  in  planning  or  evaluation  tasks. 
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Prior  to  the  initiation  of  one  of  the  quarterly  assessment  cycles  for  the  very  large  and  complex 
ADF  campaign  plan  for  Operation  SLIPPER  in  Afghanistan,  the  DSTO  team  that  was  intended 
to  support  the  task  was  comprised  of  individuals  with  varying  knowledge  and  experience. 
The  LARF  was  proposed  as  a  means  of  rapidly  exposing  team  members  to  the  entire  plan, 
while  prompting  them  to  share  and  discuss  their  interpretation  of  it.  Therefore,  in  December 
2011,  prior  to  the  2011  quarter  four  campaign  assessment,  the  LARF  was  used  for  a  one-day 
group  learning  exercise.  To  do  this  the  LARF  matrix  was  populated  with  Effects  and  then 
Decisive  Conditions  from  the  current  Afghanistan  campaign  plan  in  the  first  two  columns. 
Participants  were  then  asked  to  contribute  to  population  of  the  remaining  columns,  in  any 
order. 

This  activity  quickly  generated  a  constructive  discussion  flow  that  centred  around  the  premise 
of  the  plan  and  understanding  what  the  plan's  terminology  and  structure  was  intended  to 
mean  in  practice.  For  these  purposes,  macro-theories  were  elicited  and  debated.  This  led  to 
exploration  of  the  risks  and  assumptions  inherent  in  the  logic  for  each  component  of  the  plan 
systematically  surfaced  detail  about  the  variables  and  context  that  impact  upon  progress, 
emphasising  the  necessity  for  team  members  to  build  knowledge  on  these.  In  this  way, 
analysis  switched  from  macro-  to  micro-theories,  but  facilitators  were  able  to  monitor  the  level 
of  detail  and  raise  discussion  back  up  to  macro-level  if  participants  were  losing  focus. 
Notably,  the  LARF  process  revealed  individuals'  areas  of  expertise,  strengths  and  weaknesses, 
giving  participants  opportunities  to  request  explanation  of  aspects  of  the  plan  or  its  logic  that 
they  did  not  understand,  fostering  a  group  dynamic  of  seeking  and  offering  assistance.  This 
demonstration  of  expertise  also  led  to  a  group  notion  of  appropriate  allocation  of 
responsibility  for  portions  of  the  task.  For  a  team  of  evaluators,  interpretation  of  the  plan  into 
plain  language  was  highly  valued  because  it  gave  insight  into  what  decision-makers  were 
looking  for  as  indicators  of  success  or  failure. 

The  greater  depth  of  understanding  of  assumptions  and  risks,  also  meant  that  the  team  was 
better  placed  to  identify  information  requirements,  and  consider  what  they  should  be  looking 
for  in  those  information  sources.  Ordinarily,  some  valuable  data  sources  may  be  overlooked 
because  they  are  not  part  of  the  direct  influence  of  program  activities  or  are  not  obviously 
linked  to  the  ultimate  objectives.  However,  identification  of  interim  indicators  of  progress  by 
articulating  logic  paths  in  the  LARF,  leads  to  recognition  of  less  obvious  data  sources  that  may 
provide  some  insight,  until  conclusive  outcomes  become  visible. 

At  the  conclusion  of  the  day,  there  was  an  enhanced  sense  of  team  identity,  with  less 
experienced  members  of  the  team  having  gained  greater  confidence  in  their  understanding  of 
the  plan,  and  in  turn,  in  their  ability  to  conduct  relevant  analysis  on  assessment  data. 
Conversely,  during  this  activity  and  other  applications  of  the  LARF,  it  has  been  evident  that 
even  team  members  that  are  most  familiar  with  the  plan  and  its  context,  find  new  perspectives 
and  consolidate  their  understanding  of  linkages.  At  the  same  time,  they  benefit  from  the 
ability  of  the  inexperienced  to  contribute  fresh  insights  into  the  risks  or  assumptions 
associated  with  widely  accepted  logic.  Leading  into  the  campaign  assessment  cycle,  the  LARF 
workshop  had  produced  a  matrix  of  some  of  the  key  considerations  for  each  Effect,  providing 
a  checklist  for  evaluators  to  refer  to  when  working  through  large  amounts  of  data. 
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5.2.3  Revision  of  Plans 

Identifying  when  initial  ToC  have  been  disproven  or  need  to  be  adjusted  is  crucial  for 
adaptive  planning.  In  all  forms  of  Defence's  international  engagements,  outcomes  and  the 
conditions  that  bring  them  about  ( Decisive  Conditions)  are  complex  and  cannot  be  prescribed 
with  any  certainty.  Therefore,  we  may  need  to  be  prepared  to  adapt  our  idea  of  what  the 
Endstate  will  look  like,  as  the  operating  environment  evolves.  In  order  to  do  this  without 
losing  the  intent  of  the  initial  strategic  objectives,  planners  and  evaluators  need  to  thoroughly 
understand  them,  so  that  any  divergences  in  progress  or  alterations  to  plans  can  be  made 
within  the  scope  of  a  consistent  strategic  shape.  The  research  team's  experience  in  supporting 
multiple  revisions  and  updates  of  ADF  campaign  plans  was  that  it  is  often  done  in  tight 
timeframes,  with  any  relevant  staff  that  are  available  at  the  time,  and  with  no  pre-determined 
or  repeatable  process.  The  LARF  was  not  designed  specifically  as  a  planning  update  tool,  but 
it  does  provide  structure  for  consideration  of  all  components  of  a  plan  in  a  short-timeframe, 
highlighting  key  risks  to  the  plan's  logic  that  need  to  be  addressed,  and  is  therefore  well- 
suited  to  be  used  as  a  part  of  a  revision/ redrafting  process.  Above  all,  the  LARF  offers  a 
framework  for  examining  a  program's  ToCs  and  evaluating  whether  its  assumptions  continue 
to  hold  true,  or  whether  they  need  to  be  amended  to  avoid  the  risk  of  program  failure. 

With  this  in  mind,  in  early  2012  the  research  team  trialled  the  application  of  LARF  to  a  review 
of  the  ADF's  campaign  plan  for  operations  in  East  Timor,  in  preparation  for  anticipated  future 
requests  for  support  to  campaign  plan  updates.  This  was  carried  out  within  the  research  team, 
drawing  on  extensive  experience  with  the  campaign  plan  and  subject  matter  expertise  on 
operations  in  East  Timor.  The  LARF  matrix  was  populated  with  the  extant  plan's  Effects  and 
Decisive  Conditions,  in  columns  a)  and  b)  respectively.  From  here  the  remaining  columns  of 
the  plan  were  populated  as  described  in  other  applications  above. 

The  outcome  of  this  application  was  that  no  significant  gaps  or  duplications  in  logic  were 
identified,  and  the  manner  in  which  Effects  had  been  grouped  created  a  cohesive  macro¬ 
theory  for  how  each  Decisive  Condition  would  be  brought  about.  It  was  noted  that  there  was 
a  risk  that  only  having  two  participants  in  this  LARF  trial,  both  of  whom  were  very  familiar 
with  the  history  and  context  of  the  plan,  meant  that  much  of  the  plan's  logic  appeared  to  be 
self-evident.  The  presence  of  an  external  facilitator,  or  participants  with  less  prior  exposure  to 
the  plan,  would  have  potentially  raised  more  questions  about  the  cohesiveness  of  logic,  and 
may  have  contributed  divergent  ToCs. 

If  gaps  or  flaws  had  have  become  evident  for  a  particular  Effect  or  an  entire  grouping  under  a 
Decisive  Condition,  the  flawed  assumptions  and  unacceptable  risks  associated  with  them 
could  have  been  documented  as  explanation  of  why  planners  need  to  revise  the  plan,  and 
what  problems  they  need  to  address.  This  is  considered  to  be  particularly  useful  function  of 
the  LARF,  because  frequently,  users  of  a  plan  feel  that  it  is  flawed  or  outdated,  but  are  not 
able  to  effectively  specify  and  justify  why  it  requires  review.  The  LARF  would  allow  those 
who  have  doubt  about  a  plan  to  pinpoint  what  it  is  that  concerns  them  (i.e.  gaps,  poor 
structure,  duplication,  flawed  logic,  risky  assumptions),  and  provide  evidence  of  the  risks 
associated  with  not  addressing  those  concerns. 
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In  this  instance,  the  LARF  process  did  prompt  participants  to  reconsider  the  implications  of 
the  plan's  logic  in  practice.  The  team  were  able  to  determine  whether  previous  assumptions 
had  been  proven  or  disproven,  and  also  to  list  the  plan's  risks,  which  had  not  been  done  in 
any  systematic  way  before.  The  team  found  that  they  were  prompted  to  view  the  plan  in  the 
context  of  the  current  and  likely  future  operating  environment,  instead  of  viewing  it  as  a  static 
structure  with  insulated  internal  logic.  This  in  turn  increased  consideration  of  the  role  of  other 
Government  agencies,  and  what  communication  may  need  to  occur  with  them.  In  summary, 
this  small-scale  trial  application  of  LARF  validated  the  existing  plan,  leading  to  the  conclusion 
that  it  did  not  need  to  be  updated,  but  did  provide  additional  insight  into  the  assumptions 
and  risks  of  that  plan  going  forward. 

5.2.4  Generation  of  indicators  for  assessing  progress 

DSTO's  support  to  ADF  campaign  assessment  is  based  around  an  evaluation  framework 
which  uses  a  set  of  Measures  of  Effect  (MoEs)  assigned  to  each  Effect,  which  can  then  be 
aggregated  upwards  to  provide  an  assessment  of  progress  towards  bringing  about  a  Decisive 
Condition.  These  MoE  sets  are  indicators  of  progress,  but  it  is  not  expected  that  monitoring 
them  will  provide  an  entire  picture  of  the  program  and  all  of  the  variables  that  impact  upon  it. 
Throughout  the  provision  of  DSTO  support  to  fourteen  campaign  assessment  cycles  to  date 
for  various  ADF  operations,  LIQJOC  decision  makers  have  repeatedly  expressed  a  desire  for 
assessments  to  highlight  risks  and  to  draw  conclusions  on  the  veracity  of  the  plan  moving 
forward.  MoEs  alone  do  not  provide  this.  They  monitor  how  much  of  what  we  aim  for  is 
occurring,  but  do  not  monitor  the  extent  of  negative  trends.  Therefore,  it  has  long  been 
recognised  by  the  DSTO  team  that  the  campaign  assessment  evaluation  framework  needs  to 
incorporate  consistent  monitoring  of  risk  and  the  extent  to  which  the  broader  environmental 
context  aligns  with  the  expectations  of  the  plan. 

Following  the  trial  application  of  the  LARF  described  under  5.3  above,  it  was  confirmed  that  it 
would  be  ideal  for  drawing  out  deeper  insight  into  an  extant  plan.  Specifically,  the  micro¬ 
theory  detail  of  how  change  is  expected  to  come  about  reveals  additional  monitorable 
indicators  of  progress  that  can  be  added  to  an  existing  evaluation  framework,  as  well  as 
pivotal  assumptions  and  risks  to  be  monitored.  To  demonstrate  this,  the  LARF  was  used  in 
early  2012,  to  review  the  existing  evaluation  framework  for  the  East  Timor  campaign  plan, 
with  a  view  to  identifying  risks  and  assumptions  that  could  be  added.  This  involved  the 
research  team  working  through  the  LARF  much  the  same  as  was  done  for  the  plan  revision 
process  detailed  under  5.3  above.  The  output  for  this  application  was  identifying  perhaps  one 
or  two  key  risks  and  assumptions  that  could  added  to  the  MoE  template  for  on-going 
monitoring,  as  shown  in  Figure  6  below. 
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Effect  1:  Destabilising  elements  in  society  have  been  reduced  -  the  influence  of 
destabilising  elements  has  been  reduced  to  such  an  extent  that  they  are  no  longer  an  impediment  to 
the  sustainable  development  of  Country  X. 

Assumption:  Economic 
and  social  development  is 
an  impetus  for  the 
population  to  support  the 
legitimate  government 
and  turn  away  from 
support  to  destabilising 
elements. 

Does  this  assumption  hold  true?  [Narrative  response] 

Comment  on  assumption  [Narrative  response  -  offer  any 
thoughts  on  changes,  upcoming  events  or  indicators  which  are 
anticipated  to  verify  this  assumption] 

Risk:  Uneven 
distribution  of  economic 
development  could 
generate  further 
destabilising  elements. 

Has  the  likelihood  of  this  risk  increased  or  decreased? 

[Narrative  response  with  supporting  examples] 

Has  the  impact  of  this  risk  increased  or  decreased?  [Narrative 
response] 

Measure  of  Effect 

Criteria 

Rating 

Supporting  data/ 
examples 

Source 
of  data 

Reliability  of 
data 

1.1 

1.2 

1.3 

Figure  6:  Example  of  Measures  of  Effectiveness,  Assumptions  and  Risks  Response  Template  seeking 
input  to  Campaign  Assessment. 


As  shown  above,  those  who  provide  input  into  campaign  assessments  by  filling  out  the 
template17,  were  requested  to  make  an  assessment  of  whether  each  assumption  continued  to 
be  true,  or  whether  it  had  been  disproven  during  the  reporting  period.  Similarly,  an 
assessment  is  sought  from  each  respondent  on  whether  the  likelihood  or  impact  of  risks 
identified  for  each  Effect  have  increased  or  decreased  during  the  reporting  period. 

Another  aspect  to  using  the  LARF  for  reviewing  evaluation  indicators  is  the  validation  of 
MoEs  based  on  how  they  align  with  the  macro-theory  logic  for  each  Effect.  When  the  logic  of 
how  each  Effect  was  expected  to  bring  about  change  was  stated  in  plain,  active  language  in 
the  LARF,  this  allowed  the  research  team  to  check  whether  each  MoE  was  genuinely  linked  to 
that  logic.  In  some  instances,  it  became  apparent  that  a  MoE  was  measuring  indicators  that 
did  not  have  strong  causal  links  to  the  logic.  This  presented  opportunities  to  refine,  or  replace 


17  For  Defence  campaign  assessments,  respondents  usually  include  representatives  from  each  force 
element  of  deployed  ADF  headquarters,  relevant  Intelligence  agency  representatives,  HQJOC  Branch 
staff  officers,  Defence  International  Policy  Division  and  other  Government  agency  representatives. 
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MoEs,  to  ensure  that  assessment  data  being  sought  through  the  MoE  template  was  carefully 
targeted  and  relevant. 


6  Findings  from  Trial  Applications  of  the  LARF 

The  preceding  Chapters  have  established,  through  both  theoretical  research  and  practical 
experience,  that  effective  campaign  planning  and  evaluation  requires  a  process  for 
systematically  identifying  and  analysing  critical  information  requirements;  namely  the  logic, 
assumptions  and  risks.  The  LARF  is  designed  to  achieve  this,  and  its  viability  has  been  tested 
through  a  series  of  trials  on  current  ADF  critical  campaign  functions.  We  are  now  able  to 
reflect  on  these  trials  and  conclude  whether  it  has  been  a  genuine  enhancement  to  existing 
Defence  operational  planning  and  evaluation  processes,  and  whether  it  can  offer  additional 
benefit  with  additional  research  and  development.  This  will  be  done  by  revisiting  the  initial 
aims  of  the  research,  and  also  determining  whether  the  framework  meets  the  practicality 
criteria.  Finally,  recommendations  will  be  made  on  the  suitability  of  the  LARF  for  other 
applications,  and  what  research  may  be  undertaken  to  further  develop,  validate  and  adapt  the 
LARF  to  be  of  benefit  in  other  domains. 

6.1  Outcomes  of  Trials 

Each  of  the  four  trials  of  the  LARF  detailed  in  the  previous  chapter  are  considered  to  have 
been  successful.  In  each  instance,  the  LARF  was  easy  to  use  and  not  arduous  for  participants. 
In  each  instance  the  LARF  process  yielded  insights  and  information  outputs  that  were  not 
previously  apparent.  The  specific  value  derived  from  the  application  of  the  LARF  as 
evidenced  in  the  various  trials  is  discussed  below. 

Application  of  the  LARF  to  the  initial  planning  phase  of  the  Regional  Campaign  Plan 
demonstrated  that  it  provides  a  documented  structure  which  enables  planners  to  demonstrate 
rigorous,  comprehensive  consideration  of  all  necessary  variables.  This  in  turn  enables 
transparent  validation  of  the  scope  of  the  draft  plan,  and  the  scale  of  its  objectives,  logic  and 
planned  actions.  During  the  early  planning  phase,  the  LARF  workshop  was  also  observed  to 
provide  momentum  and  clarity  for  planners  who  are  attempting  to  navigate  their  way 
through  the  synthesis  of  a  broad  number  of  complex  variables  into  a  single  cohesive  plan. 

It  was  found  in  both  the  initial  planning  trial  and  the  planning  review  trial,  that  population  of 
the  LARF  matrix  creates  a  shared  reference  for  gaining  consensus  and  capturing  the  cognitive 
map  behind  planning  decisions,  prompting  participants  to  justify  their  version  of  logic  as 
being  derived  from  senior  directives,  or  relevant  experience.  It  also  helps  resolve 
disagreements  over  some  statements  of  logic,  where  participants  can  agree  as  long  as  a 
number  of  associated  risks  are  noted  in  the  LARF,  and  subsequently  in  the  draft  plan,  for  on¬ 
going  monitoring. 

The  LARF  is  also  effective  in  bringing  into  question  the  viability  of  some  campaign  plan 
objectives  and  effects,  by  revealing  those  for  which  there  is  no  clear  logic,  or  those  that  rest  on 
excessive  assumptions  or  risks.  As  a  result,  during  the  initial  planning  trial  and  the  planning 
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review  trial,  a  number  of  Effects  were  re-drafted,  or  moved  to  contribute  to  the  achievement 
of  a  different  Objective.  Duplications  in  logic,  or  the  apparent  risk  of  high  degrees  of 
interdependency,  also  highlighted  the  need  to  merge  or  delete  overlapping  Effects  in  both 
trials. 

For  operational  evaluation,  the  LARF  can  make  a  significant  contribution  to  the  identification 
and  verification  of  indicators  and  metrics.  This  is  evidenced  by  the  product  of  the  initial 
planning  trial,  which  was  a  matrix  populated  with  revised  Effects  and  grouping  of 
Objectives,  detail  on  the  logic,  assumptions  and  risks  for  each  component  of  the  plan,  and 
narrative  summaries  of  the  overarching  threads  of  logic,  and  the  primary  assumptions  and 
risks  of  the  plan.  This  level  of  detail  enables  evaluators  to  better  interpret  high  level  intent 
into  visible  scenarios,  for  which  milestones  and  indicators  of  success  can  be  nominated.  This 
was  reinforced  during  the  generation  of  indicators  for  assessment  trial,  where  the  level  of 
detail  provided  by  the  LARF  generated  micro-ToC,  which  were  then  able  to  be  used  to  verify 
the  existing  MoE  set.  The  process  also  resulted  generated  a  set  of  assumptions  and  risks  to  be 
monitored  in  addition  to  the  MoE  set,  which  provided  evaluators  with  tangible  indicators  of 
whether  an  MoE  had  become  redundant,  or  was  at  risk  of  not  being  achieved. 

In  each  trial,  it  became  evident  that  the  LARF  contributed  to  a  broader  understanding  of  the 
operational  context,  including  recognition  of  opportunities  and  risks  related  to  other 
Government  agency  involvement  in  the  region.  This  is  likely  to  facilitate  more  effective  inter¬ 
agency  communications  where  a  Whole  of  Government  or  multi-agency  response  is  required. 
During  the  continual  situational  awareness  trial,  benefit  was  observed  as  the  expanded 
knowledge  of  the  participants  led  to  discussion  of  new  information  requirements,  and 
consideration  of  information  sources.  Further  benefit  was  demonstrated  in  the  planning 
review  trial,  during  which  re-framing  the  plan  in  the  LARF  format  prompted  participants  to 
view  the  plan  in  the  context  of  the  current  and  likely  future  operating  environment,  instead  of 
viewing  it  as  a  static  structure  with  insulated  internal  logic.  This  in  turn  increased 
consideration  of  the  role  of  other  Government  agencies,  and  what  communication  may  need 
to  occur  with  them. 

The  process  of  populating  the  LARF  is  particularly  valuable  for  enhancing  participant's 
understanding,  engagement  and  may  improve  the  mental  model  they  take  forward  as 
planners  or  evaluators.  As  demonstrated  during  all  of  the  four  trials,  the  LARF  prompts 
planners  /  assessors  to  consider  sequencing,  temporality,  linkages,  and  interdependencies. 
For  example,  during  the  generation  of  assessment  indicators  trial,  participants  were  inspired 
to  consider  the  feedback  loops  which  exist  between  action  and  consequence  in  direct  or 
indirect  ways,  which  led  to  emergence  of  a  greater  range  of  potential  MoEs.  During  the  same 
trial,  it  also  encouraged  participants  to  look  for  indicators  of  the  opposite  state  of  achievement 
of  an  Effect  (i.e.  what  we  don't  want  to  see)  and  what  direct  and  indirect  action  may  be  a 
precursor  to  that  undesired  state  coming  about. 

One  of  the  unintentional  benefits  of  the  LARF  is  that  it  provides  an  option  for  a  constructive 
task  orientation  /  team  development  activity.  This  emerged  as  a  result  of  the  continuous 
situational  awareness  trial .  In  this  instance,  the  LARF  demonstrated  its  potential  value  for 
overcoming  the  challenge  of  establishing  a  team  across  organisations,  where  the  extent  or 
concept  of  suitable  knowledge  or  situational  awareness,  is  not  understood.  Flere ,  the  LARF 


UNCLASSIFIED 


41 


UNCLASSIFIED 

DSTO-TR-2840 

offers  a  task-oriented  mechanism  for  base-lining,  developing  and  sharing  knowledge  at  the 
initiation  of  a  task. 

6.2  Satisfaction  of  Aims  and  Practicality  Criteria 

As  a  result  of  the  trial  outcomes  summarised  above,  it  is  possible  to  make  an  assessment  that 
each  of  the  three  research  aims  defined  in  Section  1.2,  have  been  met.  Firstly,  the  aim  of 
highlighting  the  priority  information  components  of  a  plan  that  enable  CA,  has  been  met  by 
identifying  the  three  information  elements  that  are  critical  to  enhancing  understanding  of 
campaign  plans.  As  suggested  in  the  definition  of  the  Defence  CA  context,  confirmed  by  the 
literature  review,  and  as  demonstrated  during  the  LARF  trials,  visibility  of  the  logic, 
assumptions  and  risks  are  a  significant  enabler  for  effective  and  accurate  CA.  These 
information  elements  are  considered  most  effective  in  reducing  the  abstract  nature  of 
goals/ effects  by  prompting  visualisation  of  what  they  will  look  like  in  practice.  In  the  simplest 
terms,  drawing  out  logic,  assumptions  and  risks  allows  us  to  break  down  strategic  or 
operational  objectives  into  'stories'  of  how  they  are  expected  to  be  achieved,  facilitating 
accurate  prioritisation  of  activities  and  identification  of  indicators  of  progress. 

The  second  aim  of  investigating  whether  it  is  possible  to  develop  a  framework  that  can  be 
applied  to  the  CA  process  to  draw  out  the  information  components  that  have  been  identified 
as  priorities,  has  been  comprehensively  achieved.  The  LARF  fills  a  methodological  gap  in 
making  explicit  the  logic  links  between  elements  of  a  plan,  and  surfacing  underpinning 
assumptions  and  risks.  In  the  past  we  have  identified  and  monitored  risks  and  assumptions  to 
be  included  in  evaluations  essentially  via  brainstorming.  But  the  LARF  enables  us  to  do  this 
systematically  with  some  assurance  of  comprehensive  consideration  across  all  components  of 
the  plan.  It  then  enables  aggregation  of  those  assumptions  and  risks  and  visibility  of  any 
primary  risks  that  are  common  across  multiple  Effects,  which  can  be  highlighted  and 
prioritised  for  monitoring  or  prevention. 

The  third  aim  has  also  clearly  been  achieved;  namely  consideration  of  whether  the  framework 
can  be  applied  to  drafting  of  new  plans  and  reviews  of  existing  ones,  so  that  priority 
information  components  are  clearly  articulated  in  future  plans.  The  LARF  has  been  trialled  as 
a  mechanism  for  facilitating  the  drafting  of  a  plan  and  in  the  review  of  an  existing  plan.  In 
both  instances,  it  was  successful,  and  easily  implemented  within  existing  ADF  planning  and 
review  processes.  As  detailed  in  Section  6.1  above,  there  are  multiple  benefits  of  applying  the 
LARF  in  these  ways,  including  the  creation  of  a  watch-list  of  assumptions  and  risks  for 
periodic  review  as  a  systematic  means  of  updating  long  term  plans. 

The  realisation  of  these  benefits  in  practice,  has  met  the  aims  of  the  research,  but  as 
determined  in  the  research  method,  the  quality  of  the  LARF  as  a  practical  solution  is 
contingent  on  the  framework  meeting  six  practicality  criteria  for  useability  and  suitability  for 
the  Defence  operational  environment.  The  LARF  is  assessed  against  each  criterion  as  follows: 

1.  Have  clear  and  cohesive  outputs,  and 

2.  Be  easy  to  summarise  into  a  variety  of  different  formats  (ie.  summary  table;  narratives  for 
briefs  or  reports,  key  categories  for  information  requests  or  existing  evaluation  templates). 
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The  LARF  has  clear  immediate  and  subsequent  outputs,  starting  with  a  populated  matrix  of 
critical  information  requirements  for  use  in  planning  and  evaluation.  As  practical  applications 
have  demonstrated,  the  information  contained  in  this  matrix  can  be  summarised  or  expanded 
upon  for  a  variety  of  purposes,  ranging  from  cells  added  within  the  matrix  briefly 
summarising  key  issues  (See  Figure  5),  to  extraction  into  existing  evaluation  frameworks  (See 
Figure  6),  to  full  narrative  extrapolations  of  the  categories  of  information  discussed  (described 
under  5.1). 

3.  Have  a  natural  flow  that  does  not  have  participants  slavishly  stepping  through 
disconnected  components. 

While  the  framework  can  be  populated  from  left  to  right  and  its  components  are 
interdependent,  it  can  also  be  populated  in  a  different  order,  depending  on  the  knowledge 
and  preference  of  participants.  Practical  applications  have  shown  that  the  interdependence  of 
logic,  assumptions  and  risk  components  gives  rise  to  a  natural  flow  of  information,  regardless 
of  the  order  in  which  they  are  considered  (as  discussed  under  5.2).  Also,  the  matrix  does  not 
require  equal  quantities  of  information  to  be  drawn  out  for  each  component  in  order  for  it  to 
be  effective.  The  fact  that  information  may  not  be  available  for  a  particular  aspect  of  a  plan, 
can  simply  be  noted  in  the  matrix  as  an  insight,  and  need  not  hinder  the  LARF  process. 

4.  Be  flexible  and  scalable  enough  to  allow  extra  time  or  emphasis  to  be  given  to  different 
steps,  in  response  to  participant  preferences. 

The  LARF  has  been  developed  as  a  scalable  framework  that  allows  users  to  choose  which 
level  of  a  plan  they  wish  to  take  as  the  start  point  for  review,  and  which  levels  they  are  going 
to  take  as  'set  in  stone'.  This  was  in  response  to  the  fact  that  there  are  variations  in  the  extent 
to  which  plans  or  evaluation  frameworks  can  be  altered,  in  different  DSTO  support  tasks.  This 
has  been  achieved  by  considering  options  for  labels  of  each  column  of  the  LARF  matrix  which 
are  useful  for  devising  elements  of  a  plan  from  first  principles,  but  which  were  equally  well 
suited  to  categorising  elements  of  any  level  of  an  existing  plan.  Users  of  the  framework  can 
then  choose  to  review  a  plan  by  considering  the  logic,  assumptions  and  risks  of  its  highest 
level  premise;  or  they  may  not  wish  to  review  the  higher  levels  of  a  plan,  and  choose  to 
consider  the  logic  assumptions  and  risks  associated  with  lower  level  effects  and  tasks. 

As  revealed  in  the  literature  review,  both  ToC  and  program  theory  allow  for  either  micro-  or 
macro-  theories  to  be  drawn  out  and  analysed.  By  synthesising  ToC  into  the  LARF,  we  are 
able  to  use  it  to  gain  insight  into  either  the  detail  of  how  activities  and  component  parts  of  a 
plan  are  expected  to  work  (ie.  the  micro-theory),  or  to  gain  insight  into  how  the  sum  of  the 
components  is  assumed  to  culminate  in  the  achievement  of  some  form  of  change  (ie.  macro¬ 
theory). 

5.  Be  rapidly  executable  start  to  finish,  without  prior  knowledge  of  underpinning  theory  or 
techniques,  and 

6.  If  necessary,  be  executable  without  any  investment  of  resources  and  minimal  staffing. 

The  benefits  of  LARF  have  been  observed  across  a  diverse  spectrum  of  planning  and 
evaluation  phases,  for  operations  of  very  different  scale  and  context.  One  of  the  greatest 
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consistent  benefits  has  been  that  it  can  be  applied  rapidly.  This  makes  the  framework  highly 
practical  for  use  in  different  environments,  with  variable  numbers  of  participants;  either  as  a 
total  workshop,  or  as  one  of  several  steps  in  a  larger  activity.  For  example,  as  suggested  in  the 
initial  planning  trial,  the  framework  could  be  used  to  give  structure  to  a  whole  of  Government 
planning  forum.  Or,  as  proposed  in  the  planning  review  trial,  the  LARF  can  be  quickly 
applied  by  a  plan's  users  at  the  tactical  level,  to  identify  and  summarise  any  concerns  they 
have  with  a  plan  going  forward.  This  provides  a  more  effective  and  constructive  form  of 
communication  than  voicing  general  dissatisfaction,  or  relying  on  a  single  point  of  contact  to 
accurately  capture  and  convey  concerns.  The  option  defining  the  level  of  focus  at  the  start  of  a 
LARF  workshop,  as  either  macro-  or  micro-theory  (or  a  combination  of  both)  encourages 
participants  to  actively  decide  on  their  aims,  rather  than  meandering  between  strategic  and 
detailed  focus  as  the  discussion  progresses. 

6.3  Limitations 

Despite  the  achievement  of  the  research  aims  and  the  clear  success  of  the  LARF  trials,  there 
are  a  number  of  limitations  to  this  research  which  should  be  noted  in  conjunction  with  its 
findings. 

Most  notably,  due  to  the  direct  operational  support  environment  in  which  the  LARF  has  been 
developed  and  tested,  the  research  has  been  subject  to  time  constraints  and  inability  to 
conduct  empirical  testing  and  validation.  The  LARF  has  been  trialled  on  a  variety  of  tasks,  as 
opportunities  arose.  Therefore,  the  duration,  scale,  selection  of  participants,  recording  of  data 
and  assessment  of  LARF  workshops  has  been  limited  to  what  is  possible  without  infringing 
on  other  operational  tasking.  This  means  that  formal  assessment  using  pre-determined 
validation  criteria  within  a  controlled  experimental  trial  has  not  occurred. 

The  information  elicited  from  participants  for  the  LARF  is  at  this  stage  an  improvement  on 
unstructured  group  discussion  and  opportunistic  data  collection.  However,  the  means  by 
which  participant  inputs  are  captured,  documented  and  interpreted  are  the  result  of  limited 
research.  It  is  recognised  that  the  LARF  would  benefit  from  incorporation  of  expertise  in  a 
range  of  research  fields  related  to  participative  research  and  qualitative  data  capture. 

Limitations  should  be  acknowledged  in  the  suitability  of  the  LARF  as  the  solution  for  all 
planning  and  evaluation  challenges.  It  is  intended  to  be  a  solution  for  a  specific  set  of 
requirements,  and  use  in  other  contexts  may  require  adaptation  and  reconsideration  of  the 
critical  information  categories  included  in  the  matrix.  The  LARF  is  a  framework  for  the 
application  of  a  number  of  theories  to  the  Australian  Defence  operational  context,  and  while  it 
was  well  suited  to  the  practical  trials  described  above,  as  with  any  technique,  it  has  potential 
to  be  used  in  contexts,  or  for  purposes  that  are  not  viable.  Also,  while  it  has  the  potential  to 
provide  greater  clarity  and  transparency  on  goals  and  indicators  of  future  progress,  it  does  not 
enable  them  to  be  reduced  to  some  form  of  quantifiable  formula  for  success.  It  is  designed  to 
reveal  and  analyse  current  mental  models,  and  not  to  predict  likely  future  likely  sequences  of 
events.  The  data  used  to  populate  the  framework  remains  the  subjective  view  of  LARF 
participants.  Making  these  views  transparent  is  a  first  step  in  making  them  easier  to  analyse 
and  verify,  but  categorising  them  in  the  LARF  does  not  automatically  make  them 
authoritative  or  more  accurate. 
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6.4  Future  Work 

In  order  to  address  a  number  of  the  limitations  noted  above,  and  to  continue  to  develop  the 
LARF  as  an  enhancement  to  planning  and  evaluation,  a  number  of  priority  areas  for  future 
work  are  proposed. 

There  is  further  work  to  be  done  in  exploring  other  potential  applications  of  the  framework, 
including  large  scale  exercises  and  benchmarking  internationally,  across  a  variety  of  research 
disciplines,  and  agencies,  to  learn  from  the  employment  of  similar  approaches.  One  of  the 
focus  points  for  comparison  would  be  how  its  outcomes  can  be  communicated  to  decision 
makers  to  achieve  optimal  effect.  Lessons  could  then  be  learned  about  the  optimisation  of  the 
LARF  and  the  format  of  its  outputs  for  Defence  decision  making. 

Two  of  the  six  critical  campaign  functions  highlighted  in  Section  3.2,  have  not  yet  been  trialled 
as  options  for  application  of  the  LARF.  Specifically,  the  LARF  is  yet  to  be  applied  as  part  of  a 
program  completion  process  or  post-operation  evaluation,  but  given  its  demonstrated  utility 
in  fleshing  out  operational  goals  and  the  assumptions  of  what  success  will  look  like,  the 
potential  for  its  application  is  clear.  For  those  operations  where  the  LARF  has  been  integrated 
into  early  planning  or  monitoring,  it  is  expected  that  the  framework  will  be  referred  to  when 
gathering  evidence  to  determine  when  an  operation  will  be  complete.  It  is  also  expected  that 
the  detail  provided  by  LARF  for  those  same  operations  will  be  drawn  upon  for  post- 
operational  evaluation  and  lessons  capture  to  articulate  whether  in  hindsight,  operational 
goals  were  in  fact  achieved,  whether  planning  logic  and  assumptions  were  flawed,  and  the 
degree  to  which  risks  were  accurately  identified  and  mitigated.  This  hypothesis  is  an 
extension  of  ToC  where  it  is  proposed  that  having  stated  how  a  program's  activities  were 
expected  to  play  out  and  combine  to  achieve  objectives,  we  have  greater  clarity  in  hindsight  to 
ascertain  why  a  plan  or  activity  succeeded  or  failed.  Within  program  theory,  the  use  of  LARF 
for  this  purpose  would  fulfil  the  Learning  Events  component,  as  shown  in  Figure  4  above.  As 
the  large-scale  ADF  roles  in  Afghanistan,  East  Timor  and  the  Solomon  Islands  draw  to  an  end 
in  coming  years,  the  value  of  the  insights  provided  by  the  LARF  for  withdrawal  planning  and 
post-operation  evaluation  is  likely  to  become  apparent. 

Should  the  LARF  process  be  more  widely  used,  there  is  also  further  work  to  be  done  in 
incorporating  relevant  aspects  of  the  LARF  process  in  the  development  of  doctrine,  and 
various  forms  of  guidance  on  planning,  and  evaluation.  Ideally,  future  work  would  also 
include  a  greater  breadth  of  benchmarking  to  learn  whether  similar  approaches  have  been 
used  in  other  problem  domains,  and  how  lessons  from  these  may  improve  the  LARF. 

Further  work  is  recommended  to  subjecting  the  LARF  to  formal  controlled  experimental 
trialling,  with  assessment  using  pre-determined  validation  criteria.  This  may  include 
controlled  trial  validation  to  more  clearly  assess  benefits  gained  by  applying  the  LARF, 
including  seeking  formal  feedback  from  users,  and  subsequent  decision  makers.  As  a  priority, 
further  investigation  would  be  desirable  on  mechanisms  for  elicitation  of  data  from  groups 
and  individuals  to  populate  the  framework,  and  techniques  for  monitoring  the  validity  of 
such  inputs.  Ideally  future  trials  would  include  an  independent  assessor  to  record  process 
observations,  including  limitations  and  weaknesses. 
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The  LARF  was  conceived  as  a  checklist  for  drawing  out  the  types  of  information  that  have 
been  identified  as  essential  for  operational-level  planning  and  evaluation  in  Defence.  The 
mechanisms  for  drawing  out  and  using  this  information  within  the  LARF  have  been  drawn 
from  a  range  of  theories  widely  used  in  decision  support  and  evaluation  practice  outside  of 
Defence. 

This  research  has  been  successful  in  establishing  sufficient  theoretical  knowledge,  to 
combined  with  practical  experience  of  Defence's  planning  and  evaluation  requirements,  to 
develop  a  viable  framework.  Application  of  the  LARF  in  a  variety  of  trials  has  demonstrated 
that  it  has  achieved  the  research  aims.  Specifically,  the  LARF  provides  a  mechanism  for 
increasing  overall  understanding  of  the  components  of  the  campaign  plan  and  the  plan's 
comprehensiveness,  cohesiveness  and  context.  It  provides  a  framework  for  a  way  of  thinking, 
that  prompts  planners  to  ask  certain  questions  and  requires  the  articulation  of  answers  in  a 
certain  way.  It  provides  guidance  for  scientifically  considering  a  problem,  and  provides  the 
techniques  for  applying  that  mental  model  throughout  the  steps  of  planning  and  evaluation. 
This  has  been  confirmed  in  its  use  as  an  exercise  to  increase  team  situational  awareness  in 
preparation  for  a  CA  cycles,  and  as  a  tool  for  highlighting  gaps  and  flaws  for  prioritisation  as 
part  of  revision  of  an  operational  plan. 

In  all  of  the  trials,  the  LARF  proved  effective  in  reducing  the  abstract  nature  of  goals  and 
effects  by  prompting  visualisation  of  what  they  will  look  like  in  practice.  This  was  particularly 
valuable  during  drafting  of  a  new  plan,  as  it  facilitated  extrapolation  of  broad  intent  into  more 
tangible  goals.  This  enabled  planners  to  more  constructively  debate  the  achievability  of  goals, 
and  to  identify  overlaps  or  contradictions  amongst  them.  Similarly,  the  LARF  was  of  benefit  in 
plan  drafting  as  a  mechanism  for  systematically  working  through  the  plan,  indentifying  what 
progress  in  each  of  its  elements  is  expected  to  look  like,  and  what  would  indicate  a  risk  to  that 
progress.  This  benefit  was  also  experienced  in  the  other  trials,  especially  when  attempting  to 
generate  a  set  of  MOEs,  which  requires  systematic  visualisation  of  each  element  of  the  plan  to 
identify  what  the  visible  indicators  of  progress  are  likely  to  be. 

Both  DSTO  researchers  and  Defence  participants  in  LARF  workshops  have  confirmed  its 
utility  in  providing  a  check-list  of  questions  and  factors  to  consider  during  planning, 
monitoring  and  evaluation  to  improve  the  consistency  of  translation  of  strategic  intent  from 
planning,  through  execution,  to  completion  of  operations.  This  was  particularly  evident 
during  trials  for  a  draft  plan  and  for  revision  of  an  existing  plan. 

There  remains  significant  scope  for  further  development,  validation  and  application  of  the 
LARF,  particularly  in  the  use  of  data  elicited  in  the  LARF  matrix  as  a  reference  for  tracking 
whether  planning  logic  and  assumptions  were  correct  as  part  of  post-operational  evaluations. 
However,  research  to  date  has  produced  a  practical  and  immediately  applicable  framework, 
as  confirmed  by  satisfaction  of  the  practicality  criteria.  The  adaptability  of  the  LARF  and  its 
suitability  for  being  employed  rapidly,  without  a  significant  resource  liability,  makes  the 
LARF  well-suited  to  meeting  current  Defence  planning  and  evaluation  requirements,  and  it  is 
simple  and  practical  enough  to  be  adapted  for  a  wide  variety  of  future  functions. 
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